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Abstract 

This paper studies the Shannon regime for the random displacement 
of stationary point processes. Let each point of some initial stationary 
point process in R" give rise to one daughter point, the location of which 
is obtained by adding a random vector to the coordinates of the mother 
point, with all displacement vectors independently and identically dis- 
tributed for all points. The decoding problem is then the following one: 
the whole mother point process is known as well as the coordinates of 
some daughter point; the displacements are only known through their 
law; can one find the mother of this daughter point? The Shannon regime 
is that where the dimension n tends to infinity and where the logarithm 
of the intensity of the point process is proportional to n. We show that 
this problem exhibits a sharp threshold: if the sum of the proportionality 
factor and of the differential entropy rate of the noise is positive, then 
the probability of finding the right mother point tends to with n for all 
point processes and decoding strategies. If this sum is negative, there ex- 
ist mother point processes, for instance Poisson, and decoding strategies, 
for instance maximum likelihood, for which the probability of finding the 
right mother tends to 1 with n. We then use large deviations theory to 
show that in the latter case, if the entropy spectrum of the noise satisfies 
a large deviation principle, then the error probability goes exponentially 
fast to with an exponent that is given in closed form in terms of the 
rate function of the noise entropy spectrum. This is done for two classes 
of mother point processes: Poisson and Matern. 

This class of problems is of independent relevance in point process the- 
ory, where this Shannon regime has hardly been considered. Its practical 
interest to information theory comes from the explicit connection that we 
also establish between this problem and the estimation of error exponents 
in Shannon's additive noise channel with power constraints on the code- 
words. Each error exponent for the point process problem leads to an 
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exponent for Shannon's additive noise channel through some simple re- 
parameterization. When specifying this connection in the white Gaussian 
noise case in the Poisson and Matern cases, we recover the random and 
expurgated error exponents of Shannon and Gallager in the regime con- 
sidered by Poltyrev. More generally, the present paper gives new bounds 
on error exponents for Shannon's additive noise channel that hold for all 
stationary and ergodic noises with the above properties and that match 
the best known bounds in the white Gaussian noise case in the high SNR 
limit. 

Keywords: Point process, stationarity, ergodicity. Palm probability, Poisson 
point process, Matern hard core point process, stochastic geometry, random tes- 
sellation, Voronoi cell, high dimensional Euclidean space, hypersphere, informa- 
tion theory. Shannon capacity, differential entropy, entropy spectrum, typicality, 
maximum likelihood, decoding, error probability, error exponent, random cod- 
ing exponent, expurgated exponent, large deviations theory, Sanov's theorem, 
Gartner-Ellis Theorem, Laplace- Varadhan integral lemma, contraction princi- 
ple. 

1 Introduction 

This paper is focused on the capacity and on error exponents for an additive 
noise channel with codebooks consisting of a stationary and ergodic point pro- 
cess of the Euclidean space and with decoding strategies that are stationary 
relative to the underlying codebook point process. 

More general codebooks and decoding rules can of course be considered, but 
our primary aim in the present paper is to investigate what the machinery of 
stationary point processes, in particular Palm calculus, and what the simplest 
parametric classes of stationary point processes, like Poisson or Matern, can 
bring to the evaluation of capacity and error exponents. 

Since the codewords of such codebooks have no power (norm) constraint, 
the notion of capacity is not that of Shannon's additive noise channel but that 
first considered by Poltyrev in the additive white Gaussian noise (AWGN) case 
in [17] . As we shall see, the connection with Shannon's initial viewpoint (on 
capacity and error exponents for codebooks subject to a power constraint) is 
simply obtained by looking at restrictions of the point process to balls with a 
radius determined by the power constraint. All results (on capacity or error ex- 
ponents) obtained within the infinite Euclidean space, stationary point process 
setting of the present paper hence translate to results on Shannon's additive 
noise channel. 

This infinite Euclidean space, stationary point process setting was first in- 
troduced in [I], where AWGN error exponents were investigated. The present 
paper is focused on this point process setting in the case of general stationary 
and ergodic noise processes. In this case, the capacity is determined by the dif- 
ferential entropy rate of the noise process and the associated error exponents are 
derived from a Large Deviation Principle (LDP) either on the entropy spectrum 
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of the noise process or on its empirical measures. This connection between error 
exponents and Large Deviation Principles for the last two objects is the main 
new result of the paper. 

This present paper is structured in three parts. The first part (from Section 
[2] to Section m is meant to give a concise summary of the main result. For this, 
wc define notation (Section [2]), then the setting (Section [3]) and finally the main 
result (Section 13]). 

In the second part (from Section [5] to Section [5]), we define and analyze the 
Shannon-Poltyrev capacity and error exponents for general stationary point pro- 
cesses with additive displacement noise. Section [5] focuses on the definition of 
the Shannon-Poltyrev capacity. The latter is first evaluated in the white Gaus- 
sian noise case fSubscction l5.I|) . then for white noise with a general distribution 
(Subsection l5.2p and finally for general stationary and ergodic noise (Subsection 
15. 3p . Section [5] gives representations of the error probability within this setting 
in terms of the Palm probability of the underlying point process, with a special 
emphasis on the case of Poisson point processes. These representations hold in 
the general stationary and ergodic noise case (Subsection l6.3|) and admit specific 
incarnations in the white or colored Gaussian cases (Subsections 16. f I and 16. 2p . 
These representations are the basis for the analysis of error exponents. Section[7] 
gives other representations of the error probability based on the mass transport 
principle and on perturbations analysis. The Shannon-Poltyrev error exponents 
of a stationary point process are defined and studied in Section |8l The core 
result bears on the case of a stationary and ergodic noise process and assumes 
that the entropy spectrum of the latter satisfies an LDP. The error exponents 
are then deduced from an almost direct application of the Laplace- Varadhan 
lemma to the Palm probability integral representation of the error probability. 
In the present paper, we limit ourselves to error exponents associated with Pois- 
son and Matern point processes and to noise processes with densities. In order 
to provide a smooth presentation, we follow a general pattern that consists in 
looking first at the white Gaussian noise (wgn) case, then the white noise (wn) 
case, and finally at the general stationary and ergodic noise (sen) case. For 
the latter, our two main examples are the colored Gaussian noise (cgn) and the 
Markov noise (mn) case. 

The third part (from Section |9]) is devoted to the connections between the 
results of the second part and the classical results of Information Theory, namely 
the Shannon capacity/error exponent theorems for additive noise channels with 
power constraints. Subsection 19. f I recalls the classical information theoretic 
viewpoint on the capacity and the error exponent of additive white Gaussian 
noise channels in the case with restrictions on the power of codewords. The gen- 
eral connection between Poltyrev's capacity and error exponents and Shannon's 
is the object of Subsection 19.21 We show that, in the stationary ergodic noise 
case, the Shannon capacity admits an expansion when power tends to infinity, 
where the second term of the expansion is the Poltyrev capacity. This property 
is used to relate the Shannon capacity for general noise and the extension of 
Poltyrev's capacity to general noise which is given in Subsection [521 This view- 
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point allows one to retrieve error exponents which achieve Poltyrev's bounds in 
the AWGN case. It also leads to new bounds on error exponents for the additive 
noise channel with or without constraints. 

2 Notation, Basic Definitions 

Throughout the paper, all logarithms are to the natural base. We denote by: 

• -B"(a;, r) the open ball with center x and radius r in the n-dimensional 
Euclidean space M"; 

• VSi^) = f{f+i) ; the volume of B"(a;, r); 

• S'S'"~^(r) the sphere of radius r in M" centered at the origin; 

• An-i the area of SS'^-^l): A^^i = nV^{l) = r^|^; 

• Cube" (a) the cube of side length a centered at the origin in R"; 

• I T I the cardinality of a finite set T. 

Throughout the paper, for all basic definitions pertaining to 

• point process theory, see [B]; 

• information theory, see [1]; 

• large deviations theory, see [7]. 

3 Random Displacement of a Stationary Point 
Process 

3.1 Notation 

For all integers n, let 

• (KmlCn) be some measurable space; 

• M.'^{Kn) (resp. M") denote the set of simple marked point measures (resp. 
simple point measures) on R" x Kn (resp. R"); 

• A^"(A'„) (resp. Ai^) be the smallest cr-algebra containing the events 
v"'{B X L) = k (resp. v"'{B) = k), B ranging over the Borel sets of K", L 
over the measurable sets of Kn and k over the nonnegative integers (for all 
questions pertaining to measurability issues, in particular the completion 
of sigma fields, the reader should refer to [H]); 
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• MQ{Kn) (resp. Mq) be set of all simple marked point measures (resp. 
simple point measures) with an atom having its first coordinate at (resp. 
an atom at 0); 

• MQ{Kn) (resp. Mq) be the trace cr-algebra of A^"(/\„) (resp. M"); 

• (ri", J-"", P", 6*") be a probability space endowed with an ergodic and mea- 
sure preserving shift 0" indexed by t G M" . 

Each ly E M" has a representation of the form 

fc 

with ex the Dirac measure at x and {xk} the atoms of i^. To all E M", and 
all atoms x of v, one associates the Voronoi cell of point x w.r.t. v denoted by 
V{x, f) (see e.g. [IS]). Here, we take each V{x, v) an open set. 
Each /X G M"(i4r„) has a representation of the form 

fc 

with {(xfe, m^)} the atoms of /i, where Xk G K" and JTifc G Kn- The set {xfe} is 
the set of points of /i and the set {mfc} is its set of marks. 

A marked point process on R" x is a measurable map from ($7", J^") 
to (M"(A'„), A^"(i4:„)) such that for aU t G M", 

M"(0t(a.))=r,(/."(c.)) , 

where T((/i") is the translation of /i" by ~t G R". That is, if = X!fc ^T^(tj),M^(w)i 
then 

^"(^((W)) = ^e_t + rr.(„)j;^„(^) . 
fc 

We will use the following notation for such stationary marked point pro- 
cesses: 

• A„ is the intensity of /^"; 

• Rn is the normalized logarithmic intensity of /i" : A„ = e"^" ; 

• Pq is the Palm probability of /x" (by convention, under Pq , — 0); 

• is the Voronoi cell of point TJ! w.r.t. the point process i.e. = 

v(r,",M"). 
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3.2 Decoding Functional 

In what follows, we focus on specific marks of a stationary point process, which 
are regions of the Euclidean plane associated to its points. These marks are 
built from a decoding functional C", namely a measurable map from (M" x 
M", S(M") X M") to the space of open sets of R" (which can be endowed with a 
topology and can hence be made a measurable space [15|). This map is assumed 
to be such that if i/ = J2k ^ simple point measure (a codebook), then 

• For all k, 

C"(a;,i.) , Vxe V(x;!,:/) , 

with O]? an open set of M" called the decoding region of (the last 
function is not defined on the boundary of the Voronoi cell - we will not 
need this); 

• The sets {0^}k form a tessellation o/R", namely such the sets are all 
disjoint and the union of their closures is K"; 

• The mapping C" is shift-compatible, namely such that 

C"(x-t,Tt(z.)) =C"(a;,zy)-i , 

for all v, t and x. 

The decoding region of the point T^'' of the stationary point process is defined 
as 

C,"=C"(T^/."). 

It follows from the shift compatibility assumption that is a mark of point 

■ 

Below, to the stationary point process z^" = ct^ and the decoding func- 
tional C", we will associate the marked point process /i" = X]fc £t^,(-DJ,Cj); 
where, for all k, is an independent random vector of K", called the dis- 
placement vector of point TJ^ (the default assumption is that the displacement 
sequence {D^}k is i.i.d. and independent of the marked point process {T^, C^}fc) 
and is the decoding region of point TJ^. 

Each point of v'^ may be thought of as a codeword. When this codeword 
is transmitted, the channel adds to it the displacement vector so that the 
received signal is = TJ^ + _DJ! . The decoding strategy associated with C" 
expects that when TJ} is transmitted, then the received signal lands in C^. An 
error happens if this is not the case. 

The canonical example to keep in mind, which is motivated by the AWGN 
channel, is when the vectors associated to the individual points of the process 
are independent and identically distributed (i.i.d.) zero mean Gaussian random 
vectors each with i.i.d. coordinates and independent of the points. Then the 
decoding region of a point is its Voronoi cell in the realization of the point 
process [T6] . 

The most general setting concerning the noise (or displacement vectors) will 
feature a real-valued, centered, stationary and crgodic stochastic process {2?;}; 
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and displacement vectors D]? independent of the point process, i.i.d. in fc, and 
with a law defined by Dq = (2?i, . . . , 2?„) for all n. As we shall see, more elabo- 
rate though natural decoding functionals/regions then show up, determined by 
the law of {I?;};. Within this general setting, for all n, when /x", C" and the 
law of D" are given, we will define the associated probability of error as 

Pein) = hm S — (1) 

and the probability of success as ps{n) ~ l—pe{n). The limit in ([TJ exists almost 
surely and it is non-random. This follows from the assumption that the marked 
point process /i" with marks (Z?j!,CJ?) is stationary and crgodic. The pointwise 
ergodic theorem implies that 

p,{n) = i Co") = K (D- i O . (2) 

As we shall see, Pein) also has the following equivalent representation: 

PeW = [j2lY^^ecs I = E ^T^+Dj^ecs ] , (3) 

\k^0 J \fc/0 / 

(this is proved in Theorem 3.2-sen below). 



4 Main Results 

Consider the general situation where the noise is defined from some stationary 
and ergodic process V. Assume that: 

1. for all n, D" admits a density /" on M" with a well defined and finite 
diff'erential entropy; 

2. {I?/c}fc admits a finite differential entropy rate /i(I?). 

For all stationary and ergodic point processes of normalized logarithmic 

intensity — /i(2?) — ln(a) and all jointly stationary decoding regions C" = 

let 

Pe(n,M",C",a,I?) (4) 
denote the probability of error associated with these data, as defined in 



Capacity. Then, our main capacity result (Theorem 1-sen and 2-sen below) 
is that 

• For all sequences of point processes with normalized logarithmic intensity 
—h{V) — ln(a), a < 1, and for all choices of decoding regions 

lim pein,n'\C'\a,V) = 1; 
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• For all a > 1, there exists a sequence of point processes, all with normal- 
ized logarithmic intensity — — ln(a), and jointly stationary decoding 
regions such that 

lim pe{n,^i",C",a,V) = 0. 
We write c(I?) for —h{T>) and call it the capacity of the noise process V. 

Error exponents. Let 

denote the infimum of pe(n, C", a, I?), where the infimum bears on all sta- 
tionary and ergodic point processes /i" with normalized logarithmic density 
—h{'D) — ln(a). Consider the following error exponent: 

r]{a,'D) ^ liminf hi{pe.opt{n,a,'D)). 

— n n 

Our main result error on this exponent (Theorem 4-sen below) states that if, in 
addition to Assumptions 1. and 2. above, 

3 the entropy spectrum of Z?", namely the law of the random variable 
— i ln(/"(£'"), satisfies a Large Deviation Principle with a good and con- 
vex rate function /(x), 

then for all q > 1, 

v{a,V) > mi{F{u)+I{u)} , 

where 

F{u) = (ln{a) + h{V) - sup(s - I{s)) 

This result is based on an analogue of random coding, namely on Poisson point 
processes. An extension of this result to the analogue of expurgated error expo- 
nents is also given in terms of hard exclusion Matern point processes (Theorem 
3-scn-Matcrn). 

Connections with Shannon's error exponents. Let T) be some stationary 
and ergodic noise satisfying the above assumptions and with variance a^. Let 
Cp(T>) denote the Shannon capacity of the additive noise channel with power 
constraint P on the codewords and with noise T>. Let £_ (i?, P, T>) denote Shan- 
non's lower error exponent for this channel and for rate R (see Section [9] for 
precise definitions). Our main result on the connection between Shannon's er- 
ror exponent and the exponent defined above is given in Theorem 5-sen, which 
implies that for all a > 1, 

£fcpiV)-\nia)-^ln(l + ^Yp,v] > vi^^-^V) , 



where 



7y(a— ,2?) = lim 77(a — e,2?) . 



8 



5 Additive Noise Capacity of a Stationary Point 
Process 



5.1 White Gaussian Noise under MLE Decoding 

Let /i" be a sequence of stationary point processes of intensity A„ in R" . The 
displacement vectors are assumed to be independent of the point processes. 
Each displacement vector is assumed Gaussian, with i.i.d. coordinates having 
mean zero and variance . The displacement vectors are assumed i.i.d. from 
point to point. Further, we take here = V^!', the Voronoi cell of w.r.t. 
the point configuration /i". It is easy to see that this corresponds to MLE de- 
coding. The generalization of Theorem 1-wgn to arbitrary decoding functional 
as defined in Section [3. 2 1 will be a consequence of Theorem 1-wn in Section [5.21 
The main results in this case are: 

Theorem 1-wgn. For all stationary point processes such that for some sub- 
sequence Uk — > oo, liminffc_>.oo Rn^ > ^ l^. ^^^^^ ; the probability of error defined 
in (QP, for MLE decoding, is such that linik^oo Pe{nk) = 1- ^ 

Theorem 2-wgn. Let fi" be a Poisson process of intensity A„ = e"^" . For 
all subsequences Uk — >■ oo such that limsupf.^^ Rnf, < \ In 2^ea-^ ' have, for 
MLE decoding, limfc_j.oo Pe('T-fc) =0. □ 



Proof of Theorem 1-wgn. Consider the ball of radius y/nLn with the same 
volume as the Voronoi cell of point 0; namely L„ is the r.v. such that 

V^^(V^L„) = Vol(VJ). 

By an immediate monotonicity argument using the shape of the Gaussian den- 
sity. 

But 

{DJ^ i i3"(0, V^i„)) = P" E D^^iif > Ll^ , 

with D^{i) the i-th coordinate of vector I?J!. By the strong law of large numbers 
(SLLN), for ah e > 0, 



with r]f{n) — >-„^oo 0. Hence for aU e > 0, 

= 1 - K {vSiVnia^ - e)) < Vol(V;)) - r;,(n). 
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Now by Markov's inequality, 

EO(Vol(Vo")) 



(vol(VJ) > VEiVnia^-e))) < 



VBWn{a^^e)) 
and by classical results on the Voronoi tessellation, 

E°(VoTO) = ^ = e-"«". 

Hence 

Pe{n) > 1 - ?ye(ri) 



so that, in order to conclude the proof, it is enough to show that the last fraction 
tends to as n tends to infinity, for some e > 0. 

Under the assumptions of the theorem, there is some c with ct > 2c > such 
that, for all sufhciently large fc, with n = rik, we have 

^-nR„ ^ ln27re((T-2c)^ 

In addition, for all < e < 2ca — , 

for n large enough. Hence 

e""-"" /a -2c 



< I I ^ri^oo 0. 

(T — C 



□ 

Theorem 2-wgn follows from the analysis or error exponents (Theorems 4- 
wgn-Poisson and 4-wgn-Matern below). Its proof is hence skipped. 



Remark 1. 

• These theorems are linked to Poltyrev's capacity theorem [T7]. 

• There exist stationary ergodic processes which are not capacity achieving 
for WGN, namely for which the property lim supj.^02 R„,. < 5 27re3^ 
not sufficient for the probability of error to tend to with n. For instance, 
consider a stationary cubic grid with points at 

U + k{l)e-"eil) + ■■■ + k{n)e~^e{n), 

where e(l), . . . , e(n) is the orthonormal basis of R", fc(l), . . . , k{n) range 
over the integers and U is uniform over the n-cube centered in the origin 
and with side e^^ (here R E M). The Voronoi cell of any point is the 
n-cube centered at this point and of side e~^, with volume e~"^. Hence 
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the intensity of this point process is A„ = e" . In the white Gaussian 
noise case, 



Ps{n) = 




So for aU R and a, Ps{n) — > when n — > oo. 



5.2 White Noise under Typicality Decoding 

We assume now that each displacement vector has i.i.d. coordinates with a 
general distribution, the displacement vectors being i.i.d. from point to point. 
Let D he a. typical coordinate r.v. We assume that D has a density / with well 
defined differential entropy 

h{D) = - f f{x)\n{f{x))dx. 
Jk. 

Let 

c(D) = -h{D). (5) 

5.2.1 Main Results 

Our two main results are: 



Theorem 1-wn. For all point processes /z" such that there is a subsequence 
rifc — > oo with lim inf/j_yoc> i?nfc > c{D), for all choices of decoding regions 
which are subsets of R" jointly stationary with the points and the displacements 
and forming a decomposition o/M" , we have limk^oo Pein-k) = 1- 1^ 

Theorem 2-wn. Let /i" be a Poisson point process of intensity A„ = g"^n , 
For any subsequence — >■ c» such that lim supj,_j.;^ i?„j. < c{D), it is possi- 
ble to choose decoding regions that are subsets of M" jointly stationary with 
the points and the displacements, forming a decomposition of M", such that 
limfe^ooPe("-fe) = 0. □ 



Remark 2. 

• Together these results give a kind of capacity theorem for stationary point 
process perturbed by additive noise: no point process with an error proba- 
bility tending to zero offers a rate (normalized logarithmic density) larger 
than c{D). This is related to the concept of information-theoretic sphere 
packing coined by Locliger |13j . 

• Theorem 1-wn specialized to the case of i.i.d. Gaussian noise generalizes 
Theorem 1-wgn to arbitrary decoding functionals as defined in Section 

1321 
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• The Poisson point process is capacity-achieving for white noise in that it 
allows for an error probability tending to zero for any rate smaller than 
ciD). 

We now prove these two theorems. The proofs rely on the following typicality 
sets: for all n and (5 > 0, let 



^? = |(x(l),...,x(n))eR" 
By the SLLN, 



1 " 

-Y,\ogf{x{^))-h{D) 



n 



<5\. (6) 



P^((Do"(l),...,Do"W)€-4^)->™ 1. 
Proof of Theorem 1-wn. For all decompositions {C^}fc, we have 

The second term tends to as n tends to infinity because of the SLLN. The 
first term is 

\JcsnA2 / 
It is bounded from above by 

^-n{h{D)-5)'^n (Vol(CJ)) . 

But for all translation invariant decompositions of the Euclidean space 

(Vol(C„")) = ^=e-"«". 
Hence the first term is bounded from above by 

g-n-R„g-n(/i(£')-'5)^ 

which tends to when n = nt tends to infinity under the assumptions of the 
theorem. 

Proof of Theorem 2-wn. Let {CJ^}k be the following decomposition of M": 

U M n {u,^, [{rr + Al) n {rp + Am] 
\jMn{UiiTr + Am}, 

where VJ! denotes the Voronoi cell of T^'. In words, contains 

• all the locations x which belong to the set + Ag and to no other set of 
the form Tp + A'^; 
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• all the locations x that are ambiguous (i.e. belong to two or more such 
sets) and which are closer to than to any other point; 

• all the locations which are uncovered (i.e. belong to no such set) and 
which are closer to TJ} than to any other point. 

This scheme will be referred to as typicality decoding in what follows. 
Let = /i" — eo under Pg . We use the following bound: 

Po (^0 i Q) < Po (^0 ^ -4?) + Po (^0 € A^,^lnDo - ^s) > 0). 

The first term tends to because of the SLLN. For the second, we use Slivnyak's 
theorem to bound it from above by 

V^'in'^iD^ ~A2) > 0) < E"(/i"(i??^ -yt^)) =E"(^"(-yl^)) = e"^"|y^^| . 
But 

1 > r\D'fi e A'i) = f f[f{x{i))dx{i)^ f e"^^"=ii"(^(^W»dx 
so that 

\As\ < e"''''^'"*"''^. 

Hence the second term is bounded from above by Q^-^n gn{h{D)+s) ^ Using this 
for n = Uk allows one to conclude the proof. □ 



5.2.2 Examples 

Here are a few examples of geometrical shapes for the typicality sets. 

Gaussian. In the Gaussian case, ln(/(x)) = — iln(27rcr^) — and h{D) 
i ln(27re(T^), so that 



X e 



X e 



1 1 " 1 1 

-In(2^a2) + ^^x(z)2--ln(2e7ra2) 

i=l ) 



2 2 

— na 



with X = (a;(l), . . . ,x{n)). That is Ag is the difference of two concentric balls 
of radius appr. ^/na. 



Remark 3. Assuming WGN, the decoding regions of Theorem 2-wn and The- 
orem 2-wgn do not coincide. Theorem 2-wn does not admit Theorem 2-wgn as 
an immediate corollary in the Gaussian case since the decoding regions are not 
the same. 
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Remark 4. Let us stress that in Theorem 2-wn, the property Pein) — ?>r,_>oo 
does not require that 

r^{A^ c v«) ^ 1 

as n — > oo. For example, in the WGN case, the last property does not true for 
all Poisson point processes with an intensity less than capacity. To sec this, use 
the fact that x belongs to Vq iff the open ball _B"(.t, |a;|) contains no point of 
/i". This implies that 

Fo(^?cVj) = Po"(n.e5"-Mv^-VT+2?){A^"(^"(-^'I^I)) = 0}) 
= Po" (m"(-B"(0, 2V^aVT+2S)) = l) 
= P" (^"(B"(0,2V^a\/rT2^)) = o) 

where the penultimate identity follows from Slivnyak's theorem. It is easy to 
show (see Equation (|131[) in Appendix 110. 3p that if A„ = e'^^, where R = 
h In T — T— T for a > 1 , then 



c 



2VI + 2S 



< XnVS{2V^iaVl + 2S)) < 



C /2^/^T25 

\/n + 2 \ a 



for some constants C, and C". Hence if a > 2\/l + 2d, 

KiA^ c Vo") ^ 1 

whereas if a < 2^1 + 26, 



'^(^^ c v^') ^ 



as n — >■ cxj. 



Symmetric Exponential. Consider the case where D follows the symmetric 
exponential density 



/(a;) = ^ exp 



This law is centered and with variance a and E{\D\) = cr/y/^. The differential 
entropy of this density is h(D) = \n{\/2ea) and since ln(/(a;)) = — ln(\/2cr) — 

\x\V2 



A'i = |(x(l),...,x(n))eR" 
= l{x{l),...,x{n))eR" 



ln(V2<T) + — y \x(i)\ - ln(V2ecr) 

T7/T ^— ' 



<6 



V2 



< nS 



V2 



i.e. is the difference of two concentric Li balls of radius appr. n<T/\/2. 
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Uniform. Let D be uniform on [— \/3cr, +\/3(t], which is again centered and 
with variance a^. Its differential entropy is h{D) ~ hi(2-\/3cr). Hence 



-4? = Ix^ 



= {x ^ 



1 " 1 

ln(2^^^) - - E li^(l|.(.)|<y3 J - ln(2%/3a) < 5 
1=1 ) 

\x{i)\ < VSa) for all i| 



i.e. A^^ is the n-cubc of side 2\/Sa. 

5.3 Stationary and Ergodic Noise under Typicality De- 
coding 

A natural generalization of both § 15.11 and § 15.21 and is that where the dis- 
placement vectors are independent of fi" and i.i.d., but the typical displacement 
vector £)" has coordinates which form a stationary and ergodic (rather than 
i.i.d.) sequence of real-valued random variables. As already mentioned, we as- 
sume that for all n, £>" = ... ,!?„), with a real-valued, centered, 
stationary and ergodic stochastic process. Wc assume displacement vectors D" 
with a density /" admitting a differential entropy rate h(T>) and we define 

c{V) = ^h{V) (7) 

Here are two examples. 

• Colored Gaussian noise (cgn). Assume that {T'fe} is a regular sta- 
tionary and ergodic Gaussian process with spectral density function g{l3), 
i.e. 



for all k. Let r„ be the n x n matrix with entries 

Let /" denote the Gaussian density on R" with mean and covariance 
matrix r„, i.e. 



rix") = , . -, — TTTT exp — (x")T-^a;" . (8) 

^ ^ ' (27r)"/2Det(r„)i/2 2^ ^ " / ^ ' 

It is well known (see e.g. [1^) that the differential entropy rate of such a 
stationary process, namely 

h{V)^- lim - / /"(a;")ln(/"(x"))da;", 

n— s-oo n Jjj„ 

exists and is given by 

h{V) = l\n(^2e7TexpQ- 1 ln(5(/3))d/?^ ^ 

= ^ j\n{2engmdP. (9) 
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• Markov noise (mn). Here we assume that {'Dn} is a stationary Markov 
chain with values in R, stationary distribution 7r(a;)rfa; with mean and 
with transition kernel P{dy \ x) = p{y \ x)dy, where p{y | x) is a density 
on M. Then, when assuming that (2?i,I?2) has a wcU defined differential 
entropy, we get that 

h{V) = lim -is(ln(^(2?i)P(2?2|2?i)---P(2?„|2?„-i))) 
= -E{\n{P{V2\V,))) 

TT{x)p{y I x)\n{p{y \ x))dxdy /i(2?2|2'i) , 
with h{U\V) the conditional entropy of V given U. 



Let 

A^,=\x- = {x{l),...,x{n))e 



-\og{r{x^^))~h{V) 
n 



<5\. (10) 



Theorem 1-sen. For all point processes fi" such that there is a subsequence 
rifc — > cxD with liminffc_j.oo ^nfc > c(I?), for all choices of decoding regions 
which are subsets of M" jointly stationary with the points and forming a decom- 
position o/M", we have limk^ac Pe(j>-k) = 1. □ 

Theorem 2-sen. Let /i" be a Poisson point process of intensity An — e"^" . 
For any subsequence rife — > oo such that lim s\ip ^ Rn^ < c{'D), it is possi- 
ble to choose decoding regions that are subsets of R" jointly stationary with 
the points and the displacements, forming a decomposition of M", such that 
limk^ao Pe{nk) =0. □ 



Proofs. The proofs are verbatim extensions of Theorems 1-wn and 2-wn when 
using 



• typicality sets ((10)) in place of ([6]); 

• the ergodic assumption on the underlying displacement process and the 
pointwise ergodic theorem in place of the SLLN. 

□ 



Notice that in the Gaussian case, 
1 



(a;")T-^a;"-l + (i(?i) 



< 25, 



with 



din) = i ln(Dct(r„)) - (^i- Hgmdp^ 



The last limit follows from the Grenander-Szego Theorem ([TO]). 



(11) 
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6 Palm Probability Representations of the Er- 
ror Probability under MLE 



The setting is that of Seetion [3] The following representations of the error 
probability will be used for the evaluation of the error exponents in the next 
section. 



6.1 White Gaussian Noise 

Theorem 3-wgn. For all stationary and ergodic point processes and all i.i.d. 

white Gaussian displacement vectors, the probability of success under MLE ( or 
equivalently when Cq is the Voronoi cell of w.r.t. pL^) is 

Psin)= J I P^(^"(B"(rf;,r)) =0)^d^dr , (12) 

with Pq the Palm probability of /i" and with 

5^(r) = W-^^-;^^^. (13) 
J/ /i" is isotropic (invariant by rotations), then 



pM) = J K{t^"iB'\rv,r)) = 0)g^(r)dr , (14) 

r>0 

where v is any unit vector of M" . // /i" is a Poisson point process, then 

POO j'OO 

Ps{n)= e-^"^«"M5^(r)dr= / e-^"^B('''')gl'(r)dr , (15) 
Jo ^0 

with V^{r) the volume of the ball B"(0, r). □ 



Proof. Using the faet that x belongs to the Voronoi cell Vo"^ of point iff the 
open ball B"{x, \x\) contains no point of /i", we get from ([2]) that 

Ps{n) = r^{D^eVS)=P^{fi"{B"{D^^,\D^^\))^0). (16) 

Since Dq is a Gaussian vector with i.i.d. components, its norm I-Dq I has for 
density g^{x) = 5"(x/cr)/cr on R+ (see Appendix 110. ip . In addition, given that 
|Dg I = r, the law of the angle is the uniform law on SSn-i- 
In the Poisson ease, from Slivnyak's theorem, 

P^(/^"(B"(™,r)) = 0) = P"(/i"(B"(™,r)) = 0) = e'^^^Sir)^ 

□ 
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6.2 Colored Gaussian Noise 

In the colored Gaussian noise case, consider the orthonormal basis which diag- 
onahzes the covariance matrix r„. In this basis, 

n 
i=l 

i.e. the coordinates of the noise are independent and coordinate i is Gaussian, 
centered and with variance af with ct^ the z-th eigenvalue of r„ . 

Let 7?.(/x"') be the image of the point process /i" by this change of basis. 
Clearly, 7?.(^") is also stationary, but its law may be different from that of 

For x" and ?/" two vectors of M", let 

n 

Let f"(a;",r) denote the ellipsoid: 

r) = {y" e M" s.t. y") < r} . (17) 

Note that the volume of this ellipsoid only depends on r and we will denote it 
by V^{r). We have: 

Theorem 3-cgn. For all stationary and ergodic point processes /i" and all 
colored Gaussian displacement vectors, the probability of success under MLE 
(i.e. that with Cq equal to the Voronoi cell of point w.r.t. /i" and for the 
distance d) is 

Ps{n)^ f Q°(7^(M")(f"(x^d(x^0)))=0)^4===dx" , (18) 

where denotes the Palm probability w.r.t. 72,(/i"). If is isotropic, then /Lt" 
and TZ{fi") have the same law so that 

Ps{n) = / P^(M"(£:"(a;", d(x", 0))) = 0) rr ^=^dx- . (19) 

If /i" is Poisson, then 

Ps{n)^ [ exp{~X„V,"{r))g^{r)dr , (20) 

Jr>0 



where the last function is defined in \T3j). □ 
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Proof. The first property follows from the independence of the noise and the 
point process and from the MLE decoding rule which states that when a;" is 
received, one returns the codeword 

argmiUfe d(x",T,"). 

So the probability of success (given that is sent and that the additive noise 
is x") is the probability that /x" has no point 7^ such that d(a;",T^) < 
d(x",0). But 

d(:E",Tfe") >d(x",0), Vfc^O ^ /i"(£"(2;",d(x",0)))==0 

and this concludes the proof of the first statement. 
If the point process is isotropic, then Qg = Pg . 

For the Poisson case, we use ([T9|) and then Slivnyak's theorem and station- 
arity, which give that for all such that c?(a;"',0) = r, 

P;j(/i"(£"(x",r)) = 0) = P"(/i"(f"(x",r)) = 0) = P"(M"(f "(0, r)) = 0). 

The proof is then concluded when noting that the density of d{D^,0) is g". □ 

6.3 Stationary and Ergodic Noise under MLE 

We consider here the case alluded to above where the noise has its coor- 
dinates which are the n first values of a discrete time, stationary and ergodic, 
real-valued stochastic process {I?fc}fcez- We assume that the latter is centered 
and we assume that D" has a density for all n denoted by /". Wc will call 
stun of /" the function y" S M" — — ln(/"(y")) € R, which measures one's 
astonishment of getting the sample y" when sampling from /". We define the 
stun discrepancy of s" € R" w.r.t. e M" as 

D(s",t") = -iln(/"(s" -t")). (21) 
n 

In the Gaussian case, D = A' + d^/n, where d is the distance defined above and 
is a constant. Note that D is not a distance in general. In particular it is 

neither symmetrical nor positive in general. 

By definition, under MLE decoding, when x" is received, one returns the 

codeword 

argminfe D(x",T^') , 

provided the latter is uniquely defined (we will see below what happens if it is 
not the case). 

Given that = Tq is sent and that the additive noise is x", a sufficient 
condition for MLE decoding to be successful is that /i" has no point 7^ 
such that /"(x" -T^") > /"(x") or cquivalcntly such that ©(x", T^") < D(a;", 0). 
But for aU x", 

D(x",rfe") > D(x",0), V/fc 7^ ^ (m" -eo)(F(x")) = 0, (22) 
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with 



F(a;") = {y" € R" s.t. D(a;", y") < D(a;", 0)} 



{y" e M" s.t. - - Hr{x^ - y")) < -- ln(r(a;"))} . (23) 
n n 



Hence 



Ps{n) > PS f]mD",T^) > D(Z?",0)} - PS ((Ai" - eo)(F(i?")) = 0) 
\k^o J 

(24) 

and 

Pe{n) < PS (Ufe^o{B(i^", T^') < DiD\ 0)}) P^ ((m" - eo)(F(i^")) > 0) . (25) 

Also notice that the volume of the set F(x") only depends on D(a;",0) = 
— i ln(/"(x")): if this last quantity is equal to u, the associated volume is 

Vol {y" e M" s.t. - - ln(/"(a;" - y")) < u} 
n 

= Vol {y" e M" s.t. - - ln(/"(-y")) < u} 

n 

= Vol {y" e R" s.t. - - ln(/"(y")) < u}. 

n 

Hence this volume is then 

W^(u) = Vol {y" e R" s.t. - ^ln(/"(y")) < u}, m G R U {oo}. (26) 
Notice that we have 

Vol(F(x-")) = ln(r (x"))^ . (27) 

Theorem 3-sen. For all stationary and ergodic point processes /i" and all i.i.d. 

displacement vectors, the probability of success under MLE is 



Ps{n) > / PS((/i" - eo)(F(x-")) - 0)r(x")dx" . (28) 

If is such that, under PJJ, the point process yu" — eo admits an intensity 
bounded from above by the function (?"(.) on M.'^ , then 



Pein) < J min I 1, J lD(.",^")<D(.",o)5"(2/")'i2/" | r\x^)dx^ . (29) 

// /i" is Poisson of intensity X„, then 

Ps{n)> [ exp(-A„W^5(u))p?,(du) , (30) 

where p^{du) is the entropy spectrum of f" , namely the law of the random vari- 
able -iln(/"(i:>")) on R. □ 
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Proof. The arguments for proving the first and last relations are as in the 
previous particular instances of noise processes. In order to prove (|29p . we use 
(^5)) to write: 

Pe{n) < P^,{{^^--eomD-))>0) 

< min(l,E«[(M"-6o)(Fpo"))]) 



< 



/"(a;") min |^1,E[J 
/"(a;")min (l 



^O{x",T^)<B{x",0) 



1d(x",!/")<d(x",o) 9"'iy"')dy"' dx"- , 



where we used the assumption on the intensity of /i" — eo under Pq to derive 
the last line. □ 



Terminology, Key Definitions and Properties. 

1. The real- valued random variable ln(/"(Z?")) is referred to as the nor- 
malized entropy density of Z?" [llj . 

2. The law of this real- valued random variable, p^{du), is referred to as the 
entropy spectrum of 13" [llj . Notice that: 

• The existence of a density for 2?" does not imply that p^{.) admits a 
density. The latter holds iff P"(-i ln(/"(i:>")) e B) = for all Borel 
sets B with Lebesgue measure. For instance when /" is constant 
and positive on a set of positive Lebesgue measure, the last condition 
does not hold. 

• The support of p^{.) is not necessarily the whole real line. For in- 
stance, in the WGN case, the support is [1/2 ln(27r(T^), oo). In the 
case white uniform noise (see Section 15.2.21) , the support is the sin- 
gleton {ln(2^/3a)}. 

3. The sets 

SUu) - {y" e K" s.t. - - ln(/"(-y")) <u}, ueR, (31) 

n 

will be referred to as the stun level sets of 13". 

4. The volume W^{u) of S^{u) will be referred to as the stun level volume 
for u. 

5. The measure on K. defined by: 

wliB) = Vol{y" e E" s.t. - - ln(/"(y")) € B}, (32) 

n 

for all Borel sets B of the real line, will be called the stun level measure. 
The following properties hold: 
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• The measures and are equivalent. Indeed, we have 

which implies that for all bounded Borel sets B, 

g-nsup(S)^„^^) <pn(^B) < e-"'"f('^)w^(B) (33) 

Each measure is hence absolutely continuous w.r.t. the other (the 
absolute continuity condition is immediate for bounded sets in view 
of the last bounds and is then easily extended to arbitrary sets). The 
Radon-Nikodym derivative 4—?- at s g R is e"* and 4^ at s is e""'*. 

• The measure wl^ is a-finite. This immediately follows from the last 
bound. 

• For all u 

WUu)^ [ wl{ds)^ I e^'^pUds). (35) 

-/( — oo,u] -^( — 00,11] 

This is a direct corollary of the value of the above Radon-Nikodym 
derivative. 

6. The stun cell C^iT)) of point is defined as follows: 

£^(P) = {x- s.t. D{x\T^) < mmx^TP)} (36) 
U {x" s.t. D(x", T^') = D(a;", T,") for some l^kjCiVJ! . 

It is comprised of 

• The locations with a stun discrepancy (w.r.t. /") to smaller 
than that to any other point; 

• The locations with an ambiguous stun discrepancy (this includes 
the case where D(a;",T^) = c» for all k) but which are closer to 
for Euclidean distance than to all other points of p". 

These cells form a decomposition of the Euclidean space which will be 
referred to as the stun cell decomposition w.r.t. the point process ^" for the 
noise 2? (more precisely D" or /"). The stun cell decomposition w.r.t. any 
WGN noise with positive variance is the Voronoi decomposition: for all 
dimensions n, for all point processes p" on M" and for all fc, £^'(WGN) = 

This definition is somewhat Gaussian-centric. Any other tessellation whose 
cells satisfy the conditions of ^ could be used in place of the Voronoi tes- 
sellation. 
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Here are a few observations on the presence of ambiguity. When there is 
no ambiguity, there is no need to introduce an exogenous tessellation. In 
addition, if 

argmin,{©(i?o",rfc")} 

is Pg-a.s. a singleton, (|24p becomes an equality as well as (|28p and (pO)) . 
A (rough) sufficient condition for 

argmin,{©p^\r;J')} 

to be a set of cardinality less than or equal to 1 is that for all a" 7^ &" e M", 
the set 

{x" e M" s.t. /"(x" - a") = /"(x" - 6")} n Support(/") (37) 
is of Lebesgue measure in R". Indeed, if this holds true, then for all 

K [ i/"(."-T,")=/"(."-T,")r(x")rfx" = o, 

so that 

/ l/.(.._T,")=/"(."-T,")r(^")da;" = 0, 

k^l 

which in turn implies that 

Remark 5. Theorem 3-scn is consistent with the previous ones. For instance, 
in the wgn case, for all S R", 

1 1 1 " 

- - ln(r (X-)) = - ln(2.a2) + ^ xr (38) 



Using this, it is easy to show that ((30|) boils down to dTSl) . This follows from 
the fact that for all w > i ln(27r(T^): 



the set S^{u) is a n ball of radius 1 /2ncr^ ^ 5 ln(27r cr^)) , so that 



W5(m) = ( ^/2na2 ( m - i ln(27rcr2) 



the random variable — i ln(/"(I?")) is equal to i lii{2TTa'^) + ^^ J2"=i 
which implies that p^{du) admits the density 



VV V 2 yy y^2na2 (^i - 1 ln(27ra2)) ' 

In order to derive (|15p from ([50]) . it is then enough to perform the change of 



variable r — \ 2na'^ i.'^^ \ ln(27rcr2)) in the latter. 
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6.4 Mismatched Decoding 

In mismatched decoding,jthe decoder has been designed for some noise V but 
the actual noise is in fact V. By the same arguments as in Theorem 3-sen, we get: 

Theorem 3-sen-mismatch. For all stationary and ergodic point processes fJ^ 
and all i.i.d. design displacement vectors T) and actual displacement vectors T), 
the probability of error under MLE satisfies 

Pe{n) < f - eo)(F(x")) > 0).r(x")dx" . (39) 

If fi" is Poisson of intensity A„, then 

Pe{n)< [ (l-exp(-A„iy^H))p|,(du) , (40) 

where p^{du) is the law of the random variable — ln(/"(Z?")) on R and 

is the stun level volume for T) . □ 



6.5 Matern Point Process under MLE 

A Matern I point process is created by dropping points from a Poisson process 
as fohows. Choose some positive radius called the exclusion radius. Any point 
in the initial Poisson process that has another point within this fixed radius 
of it is dropped (note that both points will be dropped since the first point 
will also be within the same fixed radius of the second point). This is the 
simplest type of hard sphere exclusion. For an information theorist, this is 
reminiscent of expurgation [8] and we will also use this term below to describe 
the transformation of the Poisson into the Matern point process. A variant is 
the Matern II processes, created from a Poisson processes as follows. Mark each 
point of the initial Poisson process with an independent real number chosen 
uniformly over the unit interval. Choose some positive radius. Every point 
of the original Poisson process examines the ball around it of this radius and 
survives only if its associated mark is strictly larger than the marks of all the 
other points in this ball. These processes were introduced by Matern [T3] (the 
Matern II process will not be analyzed in this paper). 

Note that neither Matern I nor Matern II are Poisson (this is because the 
thinning is not independent). However, both are isotropic. 

Below, we introduce a new class of Matern point processes meant to cope 
with the stationary and ergodic noise framework of Section 16.31 Assume for 
simplicity that f"{x") = /"(— x"). If two points S and T of the Poisson point 
process /x" are such that D(T, S) < ^ , with ^ G M some threshold, then T is 
discarded. The surviving points form the Matern- 1?-^ point process 
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Theorem 3-sen-Matern. Under the assumptions of Theorem 3-sen, the prob- 
ability of error for the Matern-V-^ point process satisfies the bound 

Pe{n) < / niin(l,A„/ lD(y",o)>c lD(x",y")<D(x",o) I /"(a;")da;". (41) 

□ 



Proof Let Pq denote the Palm probability of /I". Under Pq , the point process 
/I" — eo has an intensity bounded from above by A„lD(y",o)>5 ^it y". The result 
then follows from □ 

Notice that the Matern-WGN-^ model boils down to Matern I for the exclu- 
sion radius 

r„(0 = VwyC-^ln(27ra2) , (42) 

for ^ > iln(27rcr^). The set {y" : D(2/",0) > ^} is then the complement of the 
ball (0,r„(0) and for all G R" with norm r, the set {y" : e(x",y") < 
D(a;", 0)} is the ball 5" (x", r) . Hence 

Corollary 3-wgn-Matern. In the WGN case, 

Pe{n)< I min(l,A„Fo;(S"(0,r„(e))^nB"(x"(r),r))).g;'(r)dr (43) 



"'r>0 

with a;"(r) = (r, 0, . . . , 0) G R" and r„(.) defined in □ 
Proof The result immediately follows from ([38]), (gl]) and (gS]). □ 

In the general (sen) case, the unfortunate fact that the volume of the vul- 
nerability set (the set which ought to be empty of points for no error to occur) : 

lD(y",0)>5 lD(x",t/")<D(x",0)'^2/" 

now depends on the point x", and not only on the value of D(a;", 0) as in the 
non-expurgated case, can be taken care of by introducing the following upper 
bound 

Mi{u,C)= sup / lD(y",0)>C lD(r",y")<« dy", (44) 

x"-: D(x".0)=u J 

which only depends on D(x", 0). This quantity will be referred to as the expur- 
gated stun level volume. By the same arguments as above, we get: 
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Corollary 3-sen-Matern. The assumptions on D are those of Theorem 3-sen. 
The probability of error for the Matern-T>-^ point process satisfies the bound 



Pe{n)< min{l,XnMviu,0)plidu) . (45) 
Jues. 

□ 

Here are further bounds on A/p(u,^): 

M^iu,^) < J lD(a",0)>J lax": D(i",0)=«, D(y",K")<ti (46) 

where we note the standing assumption in this section that /"(x") = /"(— x"), 
and 



M^{u,^) < J lD(2",0)<ti lax": D(a;",0)=«, D(z"+x",0)>4 (^2:" . (47) 

The first bound immediately follows from pij) and the fact that sup J < J sup. 
The second bound is obtained in the same way after the change of variable 
_ yU _ jj-j^ This second bound can be seen as a natural extension of 
since 



WSiu) = J 

D(z' 



2"eR": D(z",0)<'U 

> 

2"GR": D(z",0)<i 



7 Further Representations of the Error Proba- 
bUity 

In this section, we come back to the general framework of Section [3] and we give 
some other Palm probability representations of the error probability: 

7.1 Mass Transport Principle Representation 

The results of this section hold for arbitrary decoding regions. 
Theorem 3.2-sen. Under the assumptions of Section\^ 

Pe{n) =Kiy 't;^+d^ (^0 ) j ■ (48) 

□ 
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Proof. From the very definition of tlic Palm probability, 

\4,T/'e[o,i]" 

= E E E -(''^■) 

yi,T/'e[0,l]" -"62" j,T"et)+[0,l] 

= E^-f E E 



with 

But for all e Z" 



Hence 



where the last relation comes from the definition of Palm probability. □ 



Notice that the last proof only requires that the sequence {D^,CJ!}k be a se- 
quence of marks of the point process /x" = er"- The independence between 
the point process and the displacement is hence not required in Theorem 3.2-sen. 

The next result requires the independence between the marked point pro- 
cess {Tfc jC^jfe and the displacements It uses the decoding functional 
framework of Section 13.21 For all point measures i/ on R" and all x such that 
a; belongs to the support of let C"{i/) = C"(a;, i^) be the decoding region of x 
in the codebook v. 

Theorem 3.2-sen-Poisson. // yu" is a Poisson point process of intensity A„, 
then 

Pe{n) = A„ / {D^ e + dx . (49) 

□ 
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Proof. Using Slivnyak's theorem and the above notation, we have 

with A*" = X] ^t;^ ^ stationary Poisson point process. We now use Mecke's 
formula, which states that for all stationary point processes </) on R" with in- 
tensity A, defined on the probability space {n,J-',F,9x) and for all functions 

■u : rjo X R" with no = nn {0{o} = i}, 

E v{6^{uj),x)(j){dx) = XEl v{uj,x)dx, 

where is the Palm probability of cf). Applying this formula to the point 
process /u" and to the function 

V{UJ,X) = lDoeC-,(p"(a;)+e_,) 

gives: 



eri.ifi" + e_,))dx. 



□ 



7.2 Likelihood Ratio Representation under MLE 
7.2.1 Poisson 

Theorem 3. 1-sen- Poisson. Under the assumptions of Theorem 3-sen, if fi^ 
is Poisson, then 

PeH < A„ I P" ^Xr^W^ (^-lln(rP"))) < 1,-1 In ^t^^l^'^ < o) dy- 

y"eR" 

+P" (XnWi (--Hf^iD"))) > l) . (50) 



□ 



Proof. From 

Pe{n) < I P[J[(/."-eo)(F(a;")) >0]r(a;")rfa;" 



< / min(l,EJ{[(A*"-eo)(]F(x"))])r(a;")da;" , 
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with F defined in So, in the Poisson case, from Slivnyak's theorem, 

Pe{n) < [ min(l,E"[/i"(F(a;"))])r(x")da;" 

min(l, A„Vol(F(a;")))/"(a;")dx" 
min ( 1,A„ / 1d(2^",j/")<d(2^",o) dy"" I /"(x")da:" 



+ / lA„V0l(F(."))>l/"(^")rfx" 



A„ / P"(A„Vol(F(L>")) < 1, ©(£»", y") < ©(D", 0))^^" 
+P"(A„Vol(F(L'")) > 1). 



□ 



7.2.2 Matern 

The setting is that of Scction [6.5l By the same arguments as those in the proofs 
of Theorems 3-sen-Matcrn and 3.1-sen-Poisson, we get: 

Theorem 3.1-sen-Matern. Under the assumptions of Theorem 3-sen-Matern, 

Pe{n) < Xn J P" (A„^^^^^l_ii„(^„(,„))>^,l_^^/:n^^i^ 



□ 



7.3 Perturbation Analysis Representation 

In this section, n is fixed, and C"(i^) = C"(.t, i^) denotes the decoding region of 
X in the codebook i/. We make the assumption that the decoding regions satisfy 
the fohowing monotonicity properties: 

• for ah /^", ah z" e M.'^ and aU in the support of + e^") C 

q„(/i"). 
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• if a;" € C^„(/i") for some in the support of /.i", then for all z" such that 

These properties are for instance satisfied by the Voronoi and more generally 
the MLE regions. 

For all y" S R" , consider the following random subset of M" , defined under 
the Palm probability of by: 

= {x" e M" : e Co"(m") and S C;„ (/^" + e^n)} . (52) 

In view of the above monotonicity assumption 

Q'^r. = e M" : x" e and .t" ^ Co"(m" + ey^} . (53) 

In the Voronoi cell case, Q^n is the part of the Voronoi cell of under which 
is "eaten out" by the addition of to 
Similarly, for all x" S , let 

= {y" e M" : x" ^ Co"(/." + e,.)} • (54) 

In the Voronoi case, is the part of the ball of center x" and of radius |x"| 
which is not included in Vq\ The union of the "H^n domains when letting x" 
range over the whole Voronoi cell Vq is the so-called fundamental domain of 
Vo" (see e.g. [M]). 

Theorem 3.3-sen-Poisson. Under the above assumptions, if fi" is Poisson of 
intensity A ~ A„, the error probability pein) associated withC" and D" satisfies 
the relation 

d , . f f T,0/ n 



E^(Vo/(H^„)),r(x")dx" , (55) 



where is the Palm probability of the Poisson point process of intensity X on 
R". □ 



Proof. From the first order perturbation formula for functionals of marked 
Poisson point processes [3] , 



d_ 

d\' 



From the monotonicity assumptions, the last quantity is equal to 
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and hence to 

/ [^Do^Co{fJ^"' + ey^)'^DaeCo{fJ^")] ^2/" ■ 

The announced results then follow from independence and Fubini's theorem. □ 



This perturbation analysis method can be extended to higher order deriva- 
tives. These results might be of interest for the analysis of the so-called channel 
dispersion parameter (see [H]). 

8 Error Exponents 
8.1 Definition 

Let V be some stationary ergodic additive noise. For all stationary and ergodic 
point processes /i" of normalized logarithmic intensity —h(T>) — ln(a) and all 
jointly stationary decoding regions C" {C^j^, let 

pr(n,/i",C",a,I?) (56) 

denote the probability of error associated with these data, as defined in ([2]). 
The pp superscript is used to recall that the setting is the point process one 
described in Section [31 Let 

pZptin,a,V) (57) 

denote the infimum of pPP{n, fi"',C^^ , a, V), where the infimum bears on all sta- 
tionary and ergodic point processes /i" with normalized logarithmic density 
bounded from below by ~h{'D) — \n{a) and on all jointly stationary decoding 
regions. 

The error exponents for (a,!?) are then defined as 

v{a,V) ^ limsup--ln(p^^„pj(n,a,X')) (58) 

n 

r^[a,V) = liminf-iln(pf„p,(n,a,I?)). (59) 
— n n ■ ^ 

Assuming these are identical, we denote this common limit by r]{a,V). 

For a fixed point process family fi = with normalized logarithmic 

density —h(T>) — ln(Q;) for all dimensions n, and for fixed decoding regions 
C = {C"}„, let 

n{i^L,C,a,V) = lim --pPP(n, C", a, P) . (60) 
If the last limit does not exist, we define 

n{n,C,a,V) = limsup--ln(pPP(?7,Ai",C",a,I?)) (61) 

n— >-oo ^ 

7r(^,C,a,2?) = liminf--ln(pf (77,/,i",C",a,I?)) . (62) 
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Since for all /i and C as above, 
it follows that 

ri{a,V)>Tr{fi,C,a,V). (63) 

The aim of this section is to study the lower bounds 7r(Poi, a, I?) and 

]T_{Mat, £{T>), a,T>) defined as follows: the decoding regions are the MLE cells 
C(V) = {C"{V)}n, with C"(V) = {Cl{V)}k defined in ^ and the point pro- 
cesses are Poisson and Matern respectively. Notice that the normalized logarith- 
mic density fully characterizes the Poisson point process but not the Matern. 
The TT (Mat, £(!?), a, I?) will be defined for a particular class of Matern point 
processes satisfying the normalized logarithmic density constraint given above. 



8.2 Poisson Point Process 
8.2.1 White Gaussian Noise 



Let ^ be a sequence of Poisson processes ^" of rates A„ — e" where R = 
h In 75 — T— 2 for a > 1. 

From (|15p . and because the MLE cells are equal to the Voronoi cells in 
the WGN displacement case, we have the following representation of the error 
probability: 



(l - e-^"^B (--V^) j g'^{v^)^dv , (64) 



Theorem 4-wgn-Poisson. We have 

^(Poi,/:(WGN),a,WGN) = ( ^^J^ < ^ . (65) 

1^ i — In 2 + In a li V 2 < a < oo 



□ 



Proof. This theorem is a direct corollary of Theorem 4-sen below. We never- 
theless give the main ideas of a direct proof here and details on this direct proof 
in Appendix 110.31 
We have 

n/ I — \ — "( — ^— ln(i')+o(l)) 

5i (I'V") = e V 2 2 y >) ^ 

see equation (jl29l) and the discussion of the appendix, and 

P"(/x"(B"(wynU,0) > 0) = 1 - e-^"^B"(^'^v^) ^ g-«((lna-l„^,)++o(l)) 

Wc may therefore write the probability of error in (|64p as 

/•OO 2 
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We observe that the exponent of the integrand at v is of the form a{v) + b{v), 
where 

aiv) = ^-^-l-ln{v) (66) 
pertains to the additive noise and 

= (lna-lnu)+ (67) 
to the ball having some point. Carrying out the optimization: 

Minimize a{v) + b{v) over v > 0, (68) 
gives ^ — i — In a when 1 < a < \/2 and i — In 2 + In a when a > \/2. □ 

8.2.2 Beyond White Gaussian Noise 

The setting of this subsection is that of Section [6731 From (|30p . 

Pe{n) < f {l~exp{~X„WUu)))plidu) . (69) 

Below, in order to analyze error exponents, we proceed in three steps : 

1. We give sufficient conditions for the entropy spectrum /0^(.) to satisfy an 
LDP; 

2. We give a log scale asymptotic for the stun level volume W^(.); 

3. We use the Laplace- Varadhan integral lemma to deduce the error expo- 
nent from the LDP satisfied by p^{-). 

The definition retained below for an LDP is that on p. 5 of [7], which features 
a probability space (X, B), where X is a. topological space and where S is a 
cr-ficld which may be thinner than the Borel cr-field of X. 

1. LDP for the entropy spectrum. 

l.a. Stationary ergodic noise. 

The key assumption under which the stationary ergodic case will be analyzed 
in what follows is summarized below: 

H-sen 

1. For aU n, Z?" = (I?!, . . . ,X'„) admits a density /" on R"; 

2. For all n, the differential entropy of D", h{D^), is well defined; 
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3. The differential entropy rate of {Dk}, 



h{V) ^ lim -h{D'') 

n—^oo n 

= lim -iE(ln(rp"))) 

= lim -- / ln(/"(x"))/"(x")dx" 

n-)-oo n J]B>„ 

= lim / up^ {du) , 

n— f oo 

exists and is finite; 

4. The entropy spectrum p%{.), namely the law of the random variables 
{— i ln(/"(Z?"))}, satisfies an LDP (on the real line endowed with its Borel 
CT- field), with good (in particular lower semicontinuous) and convejill rate 
function I{x). 

A simple sufficient condition for 4. to hold is that the conditions of the Gartner- 
Ellis Theorems hold, namely that the limit 

lim -ln(E(ir{D^))-')) = G{9) (70) 

exists as an extended real number, is finite in some neighborhood of the origin, 
and is essentially smooth (see Definition 2.3.5 in [7]). From the Gartner-Ellis 
Theorem, the family of measures Pp(.) then satisfies an LDP with good and 
convex rate function 

I{x) = sup {Ox - G{0)) . (71) 



Remark 6. In order to analyze error exponents, there are other options than 
assuming that the entropy spectrum satisfies an LDP. In certain particular cases 
(see below), it is more convenient to assume that certain empirical measures of 
the noise process satisfy an LDP to deduce an LDP for p%{.) when using the 
contraction principle [7]. We will come back to this in what follows. 



l.b. White noise. 



In the particular case where the noise has i.i.d. components with density /, we 
deduce from Cramer's theorem that 

/(x)=sup {0x-ln {E{f{D)-'))), 

6 

with D a random variable with density /. A direct calculation gives that Pp(-) 
satisfies an LDP with rate function 

lix) = e{x)x - In (e (/(Z?)-«(-))) , (72) 

^Some of the results derived below do not require this convexity assumption. 
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where 0{x) is the unique solution of the equation x = (f>{9) with 

E{{d{D))-OHl/f{D))) 



m 



EiifiD))-") 



The uniqueness follows from the fact that this function is monotonic in 9 (as 
easily checked by differentiating and using the Cauchy-Schwartz inequality). 

Notice that the rate function /(.) is not necessarily a good rate function. A 
sufficient condition - assumed in what follows - is that is in the interior of the 
set 

{9 : E < oo} 

(see [7], Lemma 2.2.20). 

Here are a few examples. 

• WGN. We deduce from the results of OTH that 



.1-1 

when 9 < 1 and is equal to oo for ^^ > 1. So 

I{u) = sup(6'u-0/21n(27ra2) + l/21n(l-e')) 


1 



Hence 



u - h{D) - ^ ln(2w - ln(27rfT2)) 



-oo for u < iln(27r(T^); 



Iiu)-< ' , „ ' "(73) 



u 



\ In(2e7rcr2) _ i ln(2u - ln(27r(T2)) otherwise. 



which is a good and convex rate function. 
• White symmetric exponential noise. Here 

E{KD)-') ^ (^/2a)«E (^exp = > ^ < 1 • 

So 

I{u) = sup(^6lu-6lln(V2CT)+ln(l-6')) , 

that is 

/(.)=. 1+"° ^ for.<ln(y2a); ^^^^ 

I w — ft,(D) — ln(ii — ln(\/2<T)) otherwise, 

which is a good and convex rate function. 
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• White uniform noise. Here 



E{f{D)-'') ^i2V3a) 



so that G{e) = 0\n{2V^a) and 




if u 7^ ln(2^/3 
if u = ln(2^/3a•) 



(75) 



which is a good and convex rate function. 

Large Deviations on Empirical Measures Let us now illustrate Remark 
6 by a few observations on the fact that the LDP on the entropy spectrum may 
follow from an LDP on empirical measures, namely from Sanov's theorem since 
this section focuses on the white noise case. The interest of this observation is 
not purely theoretical as this derivation will also provide a new representation 
of the rate function I(x) as the solution of an optimization problem based on 
relative entropy as we shall see. 

Below, we will denote by S the support of /, namely the closure of {x g 
R : f{x) > 0}. Let K{t \\ </>) be the relative entropy (or Kullback-Leibler 
divergence) of the probability law T{dx) w.r.t. the probability law (j){dx) = 
f{x)dx, defined as follows: if r is absolutely continuous w.r.t. (p, then 



with r = If T docs not satisfy the above property, then K[t \ \ cf)) ^ oo. 

Note that r is absolutely continuous w.r.t. (j) iff it admits a density g such 
that g{y) = when f{y) = for a. a. y. In this case. 



With an abuse of notation we will continue to use the notation for its 
restriction to S. So we now have </> € Mi(S'), with Mi (5) the topological space 
of probability measures on S endowed with the topology of weak convergence. 
Let Aii{S) denote the Borel tr-algebra corresponding to this topology. The 
empirical measures 



are Mi(S')-valued random variables. From Sanov's theorem (see e.g. [7], Theo- 
rem 6.2.10), the laws of the random measures i^" satisfy an LDP on the HausdorfF 
space (Mi(S'), A^i(S')), with good and convex rate function K{. \\ (/>). 




(76) 
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Lemma 1-wn. Assume that the function x ln(/(x)) from S toR is contin- 
uous and bounded. Then the family of measures on M satisfies an LDP with 
good and convex rate function 

I{u) = inf K{T\\<j>) (77) 

tGMi(S): - f^ln{f(x))Tidx)=u 

= u~ sup /i(t). (78) 

□ 



Proof. Under the above assumption, the mapping 

I Hf{x))T{dx) 

is continuous from (Mi(5), A^i(S')) to (R,S(K)). The contraction principle 
shows that the real- valued random variables 

— ^ln(/(A))= / Kx^^idx) , 

with l{x) = — ln(/(a;)), satisfy an LDP with rate function (|77p . The infimum in 
((77|) is reached on the set of laws absolutely continuous w.r.t. f{x)dx, namely 
the laws with a density g such that g = if / = 0. Hence 

I{u) = inf K{T\\f). 

g: f=0 9=0, and -f^ln{f{x))g{x)dx=u 

It is shown in Appendix II . 5 1 that for all g absolutely continuous w.r.t. f{x)dx 
and with a finite relative entropy w.r.t. /, the differential entropy of g exists. 
Hence the following equality makes sense: 

Jjn{f{x))g{x)dx = j^-\n(^?l^g{x)dx + j^\n{g{x))g{x)dx 

= -K{T\\4>)-h{T) , 

so that 

I{u) ~ inf K{t\\(I))^u— sup ^{t). 

reMi(S): K(T\\<l,)+h(T)=u T.gMi(S): K{T\\4,)+h{T)=u 

□ 



Remark 7. Notice that the last lemma requires densities / which are 1) 
bounded from above and from below by positive constants, and hence with 
a bounded support (hence the WGN case is not covered); 2) continuous on S. 
Stronger versions of the contraction principle like those considered in [5] might 
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be useful for extending Lemma 1-wn to e.g. the case of densities with unbounded 
support. The extension should take the following observation into account; as 
shown in Appendix 1 10. 61 there exist densities / and g with infinite support and 
such that 1) / and g have the same support, 2) / has finite differential entropy, 
3) the relative entropy of g{x)dx w.r.t. f{x)dx is finite and 4) the differential 
entropy of g is not defined (i.e. has a positive part and a negative part which are 
both infinite. This shows that ((77|) - ((78)) cannot be extended as such to the case 
where one replaces the boundedness of ln(/(a;)) on S by the weaker assumption 
that / has a well defined differential entropy. 



I.e. Beyond white noise. 



• Colored Gaussian noise. In the colored Gaussian noise case, we deduce 
from the Grenander-Szego Theorem that the Gartner-Ellis Theorem holds 
with 

G{6) = ^ ln(2^) - 1 ln(l - (?) + ^ In (^^ j^^ Hgimp) , 

when 6 < 1 and G{9) — oo for 6 > 1. So, by the same arguments as above, 

foo iiu<i,j:jn{2ngmdP; 

[u - h{V) - \ In (2u - ^ jZ^ \n{2TTg{(3))dl3^ otherwise, ^ ' 

with 

hCD) = H^TTegmdP , 

which is a good, convex and continuous rate function. 

• Markov noise. We continue the thread of Remark 6 on the derivation 
of the LDP on the entropy spectrum from an LDP on empirical mea- 
sures. We let S" C denote the support of the measure on with 
density a{x)p{y\x) with respect to Lebesgue measure, i.e. the closure of 
{{x,y) : a{x)p{y\x) > 0}. We denote by Mi (5") the Hausdorff space of 
probability measures on S. The topology on Mi(S') is again that of weak 
convergence and we denote by A4i{S) the Borel cr-field for this topology. 
Finally, let SMi(iS') be the set of probability measures of Mi(iS') of the 
form b(x)q(i/\x) where y i— >■ q{y\x) is a probability density on the real line 
for each x G M., x b{x) is a probability density on R, and 

b{x)q{y\x)dx = b{y) . 

We assume that the Markov chain satisfies the uniformity condition U of 
Section 6.3 in [7]. Then Corollary 6.5.10 in [7j shows that the empirical 
measures 



1 



n '■ — ' 
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satisfy an LDP on the measurable space (Mi(5'), 7Wi(5')) with the good 
and convex rate function 

^(^^^fx(r||n«P) iireSM.iS) ^^^^ 
I oo otherwise. 

Here K is the KuUback-Leibler divergence of measures on , ri and T2 are 
the marginals of r, and ri ® P is the measure Ti{dx)P{x, y)dy G Mi(M^). 

Lemma 1-mn. Under the above assumptions, if the mapping {x, y) — >■ 
hi(p(y I x)) from S to M. is continuous and bounded, then the family of 
measures p^{.) satisfies an LDP with the good and convex rate function 

I{u) ^ inf K{t\\ti®P) (81) 

reSMi(S): /,(,)p(^|,)>o \'OL(p(y\x))T{dxdy)=u 
= U— sup h{T2\Tl), (82) 

TeSMi(S): K{T\\Ti®P)+h(T2\Ti)=U 

provided the last function is convex and admits an essentially smooth 
Fenchel-Legendre transform (see Defn. 2.3.5 ofjlj). □ 



Proof. Under the above assumptions, the mapping 

T ^ hi{p{y I x))T{dxdy) 
Js 

is continuous from (Mi(5), A^i(S')) to (R, S(M)). The contraction princi- 
ple shows that the random variables 

1 " 

An= Vln(p(2?,+i \Vi)) 

n ^-^ 

i—l 

satisfy an LDP on M with good and convex rate function ([5T|) . Let 

1 1 " 

X„ = ln(^(2?i)) - - Vln(P(2?„2?,+i)) 

n n ^ — ' 

i—l 

and 

1 1 " 1 

Bn = ln(^(2?i)) - - Vln(P(2?„2?,+i)) + - hi(7r(2?„+i)). 

n Ti ^ — ^ Ti 



n 

i=l 



The random variables Bn satisfy an LDP with the same good rate function 
I{u). This follows from the application of the contraction principle to the 
mapping 



T 



^ / (- \n{p{y I x)) - ln{TT{x)) + ln(7r(j/))) T{dxdy) . 



s 
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We also have Bn < Xn < An for all n. It then follows that X„ satisfies 
an LDP with good and convex rate function / (see Appendix 110. 4p . 

The optimization problem (|82l) can be rewritten under the following form: 

I{u) = inf K(t\\ti®P) 

TeSMi(S): K{T\\Ti(»P)+h(r2\Ti)=u 

= U- sup h{T2\Ti). 

r6SMi(S): K(T\\Ti(»P}+h{T2\Ti}=u 

because /i(t2|ti) is well defined on the set where the optimum is achieved 
(see the second part of Appendix 110. 5p . □ 



2. Log scale asymptotics of the stun level volumes. 
2. a Stationary ergodic noise. 

Lemma 2-sen. Assume that the H-sen assumptions hold. Then 
sup(s - I{s)) < liminf - ln(W^(u)) < limsup - ln(W^(u)) < sup(s - I{s)). 

s<u n— J-oo 71 ^i^QQ Tl s<u 

(83) 

The function 

J(ii) = sup(s- /(s)), (84) 
which will be referred to as the volume exponent, is upper semicontinuous. □ 



Proof. From §5^), 



with 

'/>(s) = Sls<u - OOls>„. 

with satisfying an LDP and since the function is lower semicontinuous, 
the lower bound is proved as in Lemma 4.3.4 in 0. Similarly, the function 

4>{s) = Sls<u - 00ls>ti 

is upper semicontinuous and the upper bound is proved as in Lemma 4.3.6 in 
[7]. In both cases, it should be noticed that the proofs in [7] actually allow for 
functions with values in {— oo} U R. 

Let us now show that the upper semicontinuity of the function ^(.s) = s— /(s) 
implies that of the function J{u) = sup^<„ ,g(s). We have to show that 

J{u) > lim sup J(s) = hm J{u + e) , (85) 
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where the last inequality follows from the fact that J is non-decreasing. Hence, 
using monotonicity again, we have to show that J is right-continuous. We have 

J{u + e) = J{u)+ sup (5(s) - J(u))+ , 

s£ [it,'u4-e] 

with a+ = max(a, 0). So, either g{s) < J{u) for all s € [m, u + e], in which case 
J{u + e\ = J{u) and the right-continuity is trivially satisfied, or g(s) > J{u) for 
some s €z [u,u + e], in which case 

J{u + e) = sup g{s). 

[u^u-\-e] 

It then follows from the upper semicontinuity of the function g{s) that 
Jiu) > g{u) > lim sup g{s) = lim J{u + e), 

so that ([55)1 and hence right-continuity hold in this case too. □ 
Here are two observations: 

• Since I{h{V)) = 0, it follows from ^ that J{h{V)) > h{V). The concav- 
ity of the function x — > x~I{x) implies that this function is non decreasing 
on the interval (— oo, /i(2?)]. Hence, from ()84p . 

J{h{V)) = h{V) . (86) 

• At all points u of continuity of J 



lim -\n{W^{u)) = J{u) . (87) 

n— ^oo 71 



2.b White noise. 



Below, we give a few basic examples and we use the following observation: 
consider the white noise case with law </> admitting the density / on R. When 
((78|) holds, it follows from Lemma 2-sen that 

sup h{T) < liminf iln(W^(u)) (88) 

t: K(T\\(t,)+h{T)<u 

< limsup i ln(Wp(w)) < sup ^{t) . 

n-i-oo t: K(T\\(t>)+h(T)<u 

• WGN. In the white Gaussian noise case, it immediately follows from 
Lemma 2-scn that 

, , f-oo for u < iln(27rcr2) 

I i ln(27recr^) -t- ^ In (2u - ln(2 TTcr )) otherwise . 
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There is of course a more direct derivation of this result based on the log 
scale asymptotic of the volume of the ball of radius y/n2a^{u — 1/2 \n{2T:a^)). 

Here is a third (heuristic) way: in spite of the fact that there is no justi- 
fication of ([75)1 in the WGN case, it is easy to see that the optimization 
problem (|88p gives the right answer too. We have 

-ln(/(x)) = iln(2^cT2) + ^ 

and (j88p boils down to finding the distribution function r of maximal 
differential entropy and with variance less than u — iln(27rcr^). This of 
course requires that u > iln(27r(T^). The solution to this optimization 
problem is the Gaussian distribution function t* with variance 2a^(u — 
iln(27rcr2)). Hence 

lim -\n{W^{u)) = h{T:) = iln(27re) + ilnf2a2(^-iln(2^a2)) 

= iln(27recr2) + iln(2u-ln(27rcr2)) . 

White symmetric exponential noise. We can again evaluate the vol- 
ume by three different ways. The first is Lemma 2-sen and gives 



for u < ln(\/2 
otherwise . 



The second is geometric: using the fact that the volume of the Li ball of 
radius a is , we get that for u > \n{\/2a), 



L" s.t. iViy.l— <ii-ln(V2a)l 
n ^ — ' a 



W^(u) = Vol 

(V2cr(M- ln(\/2cr))n)" 



exp 



(n\ii{V2ea{u - ln(%/2cr))) + o(n) 



The third (heuristic) way is based on the solution of the optimization 
problem (|88p . The latter reduces to finding the distribution with maximal 
entropy on R with mean absolute value equal to t = a{u — ln(A/2cr))/\/2. 
This is the symmetric exponential with parameter a' ~ t\/2 (see Appendix 
I10.2|) which has for differential entropy: 

ln(V2ea') = ln{V2ea{u - \n{V2a))). 

The three approaches are hence consistent. 
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• White uniform noise. From Lemma 2-sen, 

f-oo for . < ln(2y3a) 

|ln(2V3(T) for u > lii(2\/3cr) , 

which also fits with the other viewpoints. 

In the first two examples, the function u — > J{u) is continuous. In the third 
one, the function is only upper semicontinuous. 

2.C Beyond white noise. 

• Colored Gaussian noise. From ((79l) and ((84l) . in the colored noise case 



Ji^) = \ h ln(2^eg(/3))d/? + 1 In (2u - ln(2^5(/3))d/3) (92) 

otherwise . 

This function is continuous. 
• Markovian noise. 

Under the assumptions of Lemma 1-mn, 

sup hij^Vi) ■ (93) 

reSMi(R2): K(r||Ti(»P) + h(r2|Ti)<« 

3. Error exponents. 

3. a. Stationary ergodic noise. 

Theorem 4-sen-Poisson. Assume that /i" is Poisson with normalized loga- 
rithmic intensity ~h{T>) — \Ti{a) with a > 1 and that decoding is MLE. Assume 
the noise T) to be stationary and ergodic and such that the Assumptions H-sen 
hold. Then the associated error exponent is such that 

TL{Poi, C{V), a, V) > inf {F{u) + I{u)} , (94) 

u 

where I(u) is the rate function of (defined in ^71^ ) and 

F{u) = (ln(a) + h{V) - J{u))+ , (95) 

where J{u) = supj,<„(s — /(s)) is the volume exponent defined in Lemma 2-sen. 
□ 
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Proof. We use and the bound 
to write 

^ u 

with 



<^„(«) - (^ln(a) + h{V) - 1 ln(W^^(u))^ . 

In order to conclude, we use Theorem 2.3 in [20]. Since the law p^{du) 
satisfies an LDP with good rate function I{u), it is enough to prove that for all 
S > 0, there exists e > such that 

liminf inf ( ln(a) + h{V) - ^ ln(W^(it))^ > (ln(a) + h{V) - J{u)))^-S. 

n-i-oo (^u-eM+e) \ n J 

Since the function u — > (u) is non decreasing, the last relation is equivalent 
to 

liminf ( ln(a) + h(V) - - ln(W5(u + e))^ > (ln(Q!) + h(V) - J(u)))+ - S. 

n— >-oo y n J 

Hence it is enough to show that for all (5 > 0, there exists e > such that 

lim ( ln(a) + h{V) - sup — ln(H7^(w + e))] > (ln(a) + h(V) - J(w)))+ - S. 

There are two cases: if ln(a) + h{T>) — J{u) < 0, the result is obvious, if 
ln(a) + h{'D) — J{u) > 0, then we have to prove that for all S, there exists an e 
such that 

lim sup — ln(VKp (w + e)) < sup(s - I{s)) + 6 
But from Lemma 2-sen, 

lim sup — ln(W5*(u + e)) < sup (s - /(s)). 

Hence it is enough to show that for all (5, there exists an e such that 
sup(s — I{s)) > sup (s — I{s)) — 6 . 

s<~u s<'U+e 

This follows from the fact that the function J{u) is upper semicontinuous. □ 

Notice that all terms in the final expression to be minimized, namely 
{\n{a) + h{V) - J{u))+ + I{u) 
can be traced back as follows: 
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• e-('"(")+''(^)) is the intensity A„; 

• e"''^"-' is the volume of the stun level set for level u; 

• e~"^^") is the value of the density of the entropy spectrum at u; 

• the positive part stems from the minimum of the mean number of points 
in the above set and 1. 

3.b. White noise. Here are direct applications of the last results. 

• WGN. In the white Gaussian noise case, we find back the minimization 
problem (|68p and hence the result of Theorem 4-wn-Poisson from as 
follows: use the formula ([75|) for / and the formula ([5^ for J in and 



pose V = ^2{u — ^ ln(27ro'2)). 



• White symmetric exponential noise. We get from the formula (|74p 
for / and the formula (|90p for J that, in this case, the function to minimize 
in (El) is 



v-1- ln{v) + (ln(a) - ln(w))+, 

for u > 0. So in this case, 

m • /-^TiN a-l-lna ifl<a<2 , , 

2r(Poi,/:(2?),a,P)>| i_21n2 + lna if 2 < a. ^^^^ 

• White uniform noise. Here we deduce from (|9ip and (1751) and ([M)) that 

7r(Poi, C{V), a, V) > F(ln(2\/3cr)) = ln(a) . (97) 

3.C Beyond white noise. 

• Colored Gaussian noise. In the colored Gaussian noise case. Theorem 
4-scn, dZl) and IMl) give 



^(Poi, £(2?), a, V) > inf | (^ln(«) - i In (^2u - ^ J ln(2^g(/3))rf/3^ ^ 
+u - ^ y ln(2^e5(/3))d/3 ~\^'^{^^~ ^ j ln(27r5(/3))d/3^ | . 



Posing 



27r 

we get that the last infimum is 



v= hu^—l ln(27r5(/3))d/3 



inf I {Ha) - \n{v))+ + ^ - ^ - Hv) 

v>0 Z Z 
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and we hence get the same function to optimize as in the wgn case. So 
the exponent is that of formula ((65|) . 

We now give a direct analysis of this result. From (PU)) . 



(l _ g-A„vi-(i,v^) j gn(^y^^)^^dv . (98) 



In the orthonormal basis which diagonalizes the covariance matrix, we 
have 

n 

1=1 

Using the Grenander-Szego Theorem, we get that 

n 

■Q^_ g"[2^/-„ln(9(/3))'^/3]+o(l). 
i=l 

Hence if ^" is a sequence of Poisson point processes of rates A„ = e"^ 
where 

R = -h{V) - ln(a) = -i ln(27re«2) _ i_L ln(5(/3))d/3 , 
with a > 1, then 

1 _ er^■n.m=l'y^VS(vV^) ^ g-n((lnQ-ln«) + +o(l)) ^ 

The second term of the integrand in is the same as in the white case 
(see ((64)) ). So, the exponent a{v) + b{v) of the integrand is indeed the 
same as in the white Gaussian noise case. 

• Markovian noise. Under the assumptions of Lemma 1-mn, Theorem 
4-sen, (|H2|) and ^ give 

7r(Poi,/:(X>),a,X>) > inf |(ln(Q) + /i(X') - V(u))^ + u- 0(u)| , (99) 



with 



h{V) = h{V2\V^) 

0(w) ~ sup /i(t2|ti) 

reSMi(R2): _fS-(T||ri(g)P)+/i(T2|ri)=« 

l}j{u) = sup h{T2\Ti) 

reSMi(R2): K{T\\TifSP)+h{T2\Ti)<u 
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8.3 Mismatched Decoding 

The setting is that of Section 16.41 



Theorem 4-sen-Poisson-Mismatch. Assume that ^" is Poisson with nor- 
malized logarithmic intensity —h{'D) — hi(a) with a > 1 and that decoding is 
MLE. Assume both the design noise V and the actual noise V to be stationary 
and ergodic and such that the Assumptions H-sen hold. Then the associated 
error exponent is bounded from below by 

inf +7(u)} , (100) 

where I{u) is the rate function of p~ and 

F{u) = (ln(a) + h{V) - J{u))+ , (101) 
where J{u) ~ sup^<„(s — I{s)) is the volume exponent for V. □ 



8.4 Matern Point Process 
8.4.1 White Noise Case 

Throughout this subsection, V is WN with coordinates with density / on M. As 
above, S denotes the support of /. 

Given x" £ R" we associate to it the empirical distribution t^'^ E Mi(S') 
defined by 



1 

r,.(A) = -Vl(x, €A) 

T7 ' ^ 



n 

1=1 



for all Borel sets A of R. 

The integrand of the inner integral in the RHS of eqn. (|^T|) depends only on 
Txr^. To see this, fix x" with empirical distribution t — t^^. We discuss S M" 
via its conditional empirical distribution given x" , denoted 7 = lT,y-^i defined 
as the regular conditional distribution satisfying, 

/ / "f{dv I u)T{du) = - V l(xi e A, iMeB) . 

Then 

{D(2/", 0) > a ^ / / - Hfiv))lidv I u)T{du) > e , 



and 

{e(x",2/") < D(a;",0)} ^ / / -ln{f{u-v))j{dv \ u)T{du) < I - ln(/(u))r(du) 
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So we can write 



^O{y^,0}>i^a(x",y")<O{x",0)dy" 



as 



l/p„ - l„(/(«-«))7(dt-|n)r(d«)</., - ln{fiu))T{du)dy"- 

From Sanov's theorem, the laws of the random measures r^n satisfy an LDP 
on the space {M.i{S), A4i{S)), with the good and convex rate fmiction K{. \\ 0) 
defined in ([75]). 

So, if for aU e Mi (5), 

Ml") -^n^oo V'M, (102) 

then one can expect to deduce from Varadhan's lemma that 
7r(Mat, £(!?), a, P) > inf {{- ln{a) ~ h{D) ^ 'iP{t))+ + K{t \\ (jj)) . (103) 

reMi(S) 

This is of the same form as our main result in Theorem 4-sen-Poisson. 

Notice that the same idea can be applied to all cases where the empirical 
distribution of the noise satisfies an LDP. For instance, it can be extended to 
the Markov case by using pairwise-transition empirical distributions. 

The main technicality in both cases is the proof of (|102p . We have no general 
result on the matter at this stage. We rather look at two special cases below. 



8.4.2 White Gaussian Noise 

Throughout this subsection, V is WGN with variance and decoding is MLE. 
Fix e > 0. Consider a sequence of Matern I processes pi". The point process pt" 
is built from a Poisson processes /i„ of rate A„ = e"^ where i? = ^ In ^^^^2g.2 for 
a > I, and has for exclusion radius (a — e)ay^. The intensity of this Matern I 
point process is 

and it is easy to see that — >n-i.oo 1, with A„ < A„ for all n. 

Let 7r(Mat, £(WGN), a, WGN) denote the error exponent ([62|) associated 
with this family of Matern point processes. 

Theorem 4-wgn-Matern. We have 

7r(Mat,£(WGN),a,WGN) > — , for aU a > 2. (104) 

8 

□ 
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Proof. Take an exclusion radius of {a — t)a%/n. From Formula 

Vein) < f min(l,A„Vol(B"(0,(a-e)aV^))'nB"(y"(^;),(wyH))) 
gi{vy/n)y/ndv , 

with ?;"(«) = {va^/n, 0, . . . , 0). It is shown in Appendix 110.71 that 

Vol (B" (0, (a - e)aV^))' n S" (?;"(«), (w^n)) < VSic{v)aV^) , (105) 
with 

if < u < f 



civ)^< Jv'-iv-^y) iff<«<;% (106) 



V2 

with a = a — e. Hence, by the same arguments as in Theorem 4-wgn-Poisson, 
7r(Mat,£(WGN),a,WGN) > inf 6(u) + a(?;) , 

with a{v) — ^2 — ^ — Inw and 

{oo if < 1) < f 

In a - iln(^;2 _ (^, _ ^f) if | < „ < ^ (107) 

(In a - lni;)+ if ^ < w . 

The announced result follows when minimizing over v for each a > 2 and then 
letting e to 0. □ 



Remark 9. The lower bound on r/(a) given in the two last theorems, namely 

if 1 < a < V2 

if^/2<a<2 (108) 
if a > 2 . 

was first obtained by Poltyrev [T7| (see eqns. (32) and (36) therein) and is 
referred to as Poltyrev's error exponent. The Poltyrev random coding error 
exponent is defined to be the part of this function which is comprised of the two 
portions 1 < a < and < a < 2. So the Poisson point process achieves 
this Poltyrev random coding error exponent in this range, whereas the Matern 
I point process achieves the Poltyrev function in the range a >2. 



Tr(a) 



ln( 



- In 2 + In a 



8.4.3 Symmetric Exponential Noise 

This section is focused on the case where V is white symmetric exponential 
noise and on the Matern- 1?-^ point process, which is a hard exclusion process 
where the exclusion regions are Li balls of radius 

r"(0 = ^(e-ln(^/2a)) , 
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for ^ > ln(\/2cr)- In what foUows, we build this Matern point process, pt", from 
a Poisson point process /i" of intensity A„ = e"^ with R = — \n{^/2eaa), where 
a > 1. The parameter ^ is chosen as follows: 

C = a-e + ln(V2cr) , 

so that the Li exclusion radius is 

The intensity of the associated Matern point process is then 
with 

the volume of the Li ball of radius r„. It is easy to see that A„ < A„ for all n 

and that lim„_yoo ~ 1- 

By following the same line of thought as in the proof of Corollary 3-wgn- 
Matcrn, wc deduce from (|4T]) that 



Pein)< /min(l,A„ sup Yo\(b'^ {0,r„r n {x'\r))] g2ir)dr, (109) 



r>0 



where |.|i denotes the Li norm, _B"(.T,r) the Li ball of center x and radius r 
and gj(r) here denotes the density of the Li norm of D", which is r{9, n) with 
- ^ that is 



(J ' 



So, when posing r = vna/\/2, we get that the R.H.S. of ()109p is 

1,A„ sup ) e-™^^dz; , (110) 

y wi (n) 



r>0 

with 



a:":|x"|i = 



iy(:r",i;)=Vol(Bi"(0,^^^^^^ynSr('x",^) ). (Ill) 



Let 5 = a - e. If w < f , then W{x",v) = for all x" with |x"|i = It is 
proved in Appendix 110.81 that if w > ^, then 

lim-lnl sup W{x",v)] :^ln{V2vea) . (112) 
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So, by the same arguments as in the WGN case, the error exponent associated 
with this sequence of Matern point processes satisfies the bound: 



7r(Mat, C{V), a, V) > inf b{v) + a{v 



v>0 



with a{v) = V — \n(v) — 1, (stemming from e ^" ir"n) )' ^^^'^ 
b{v) = 



if < u < 



_ _ 2 

(In a — lnw)+ if ^ < • 
(stemming from min ^1, A„ sup^„.|2,r,|^^^ W{x"-,v)j in (|110p ). This leads to: 

Theorem 4-symmetric-exponential-Matern. For T) symmetric exponen- 
tial, we have 

{a - ln(a) - 1 for a < 2 

ln(a) + 1-2 ln(2) for 2 < a < 4 (113) 

f - ln(a) -1 + 2 ln(2) for a > 4. 

□ 



9 The Channel with Power Constraints 
9.1 The AWGN Channel 

Consider communication over the AWGN channel with noise variance cr^ per 
degree of freedom using codewords of block length n subject to the power con- 
straint P. 

We write for the signal-to-noise ratio Pja^. A code is a finite subset T 
of points in B"(0, \JnP) q The elements of the code arc called codewords. We 
call i?(T) = i In I T I the rate of the code. Consider the Voronoi decomposition 
of M" determined by the codewords. For each codeword, the probability that 
a Gaussian random vector in K", centered at the codeword with independent 
coordinates each having variance a^, lands outside the corresponding Voronoi 
cell, is called the associated probability of error. By the average probability 
of error of the code, denoted Pe(T), we mean the average over the codewords 
of the corresponding probability of error. Given T, this only depends on the 
signal-to- noise ratio A^. One of the central problems of coding for the AWGN 
channel is that of understanding, for each fixed A > 0, the set of achievable 
pairs {R{T),Pe{T)), as one varies T. 

^Another option consists in taking the codewords in SS"~^{VnP). The two options lead 
to the same results. 
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9.1.1 Shannon Capacity 

For fixed A > 0, given i? > 0, we may ask whether there exists a sequence of 
codes Tn in K", n > 1, having rate at least R, asymptoticany as n — >■ oo, and 
such that Pe{Tn) — > as n — > oo. As proved by Shannon, this is possible if 
i? < i ln(l + A'^) and not possible if i? > § ln(l + A^). Thus i ln(l + A'^) may 
be called the Shannon capacity of the AWGN channel. 

9.1.2 The Error Exponent 

Let A > and < i? < | ln(l + A^). Let Pe^optin, R, A) denote the infimum 
of Pe(T) over all codes in R" of rate at least R when the signal-to- noise ratio is 
A^. We write ^ 

Se^e,ct{n, R, A) = log Pe^opt{,n,R, A) . 

Let ^oxact(-R, ^) = linisup5oxact(«, ^,^), and 

n 

{R,A) = liminffexact(n,i?,A). (114) 

n 

Assuming these arc identical, wc denote this common limit by fexactC-R, A). For 
fixed A > the function R i-> £oxact(-R, ^) is called the error exponent or the 
reliability function of the AWGN channel when the signal-to-noise ratio is A^ . 

The best currently known upper and lower bounds for the error exponent 
are given in [5], to which we refer the reader, d We write R t-^ £sGiR,A) for 
the best known lower bound, since it is due to Shannon [19] and Gallager [8]. 
We write R i— SABLiR^ A) for the best known upper bound, since it is due to 
Ashikhmin, Barg and Litsyn [5] . These bounds coincide, and thus Ecxact {R, A) 
is known, for 

1 , A A 1 I A^\ 1 , , 

- n -H h-\/H <R< - n l + A^) , 

2 \^2 4 2V 4 J - -2 ^ ' ' 

but there is a gap between these bounds for all lower rates. 

9.2 The Additive Stationary Ergodic Noise Channel 

The setting is the same as above but for the noise vector which is not i.i.d. 
Gaussian anymore. Here, the noise vector for dimension n, D" = . . . , 2?„), 
has its coordinates which are the n first values of a discrete time, stationary and 
ergodic, real-valued stochastic process {'Dk}k>i- We assume that Vq is centered 
and has a finite variance. We also assume that D" has a density on E" for all 
n denoted by /". 

^ Note that the upper bounds are on f exact {R,A) and the lower bounds arc on f ^xact ^) ■ 
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9.2.1 The Shannon Capacity 

We assume here that the codewords belong to B"{0,y/nP). The Shannon ca- 
pacity of the additive channel with this noise and under this restriction is known 
to be 

Cp(2?)=lini- sup I{r\r' + D''), (115) 

where the supremum bears on all distribution functions for T" = (T", . . . , T") € 
R" such that E(Er=i(^")^) < "-P- 

9.2.2 Connections between Capacities with and without Restrictions 

Let (T^ denote the variance of I?(0). The relation between the Shannon capacity 
and that defined in Theorems 1-sen and 2-sen is given by the following lemma 
(which is due to Shannon - we give a proof for the sake of self containedness) . 

Lemma 5. Under the foregoing assumptions, 

i ln(27reP) + c{V) < Cp[V) < i ln(27re(P + cr^)) + c{V) (116) 

and 

Cp{V) = i ln(27reP) + c{V) + 0(1/P) , (117) 
when P tends to infinity. □ 



Proof. We have 

/(T" + P"; T") = /i(T" + P") - /i(T" + I T") = /i(T" + P") - /i(2?") 
It is well known that for all stationary sequences Ai, A2, . . . 

Ti 

/i(^i,A2,...,A„) < -ln(27reVar(^i)). 

Hence 

-/(T" + P"; T") < - ln(27re(P + d^)) - i/i(r»") . 
n 2 n 

This gives the upper bound and the fact that 

Cp{V) < i ln(27reP) - h{V) + 0(1/P) , 

when P tends to infty. 

For the lower bound, we use the inequality 



/i(r" + x>") > /i(r") 
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to deduce that 

/(T" + P"; T") = /i(r" + 2?") - /i(T" + I T") > /i(r") - ;i(2?") , 
Taking now T" Gaussian with i.i.d. N{0,P) coordinates, we get that 

Cp{V) > iln(27reP) - h{V). 



□ 



9.2.3 Error Exponents with Restrictions 

In tlic case with general noise, one defines 

£in,R,P,V) = -^\ogPe,opt{n,R,P,V) , (118) 

with Pe.opt{n, R, P, T)) the infimum of pe(T) over all codes in M" of rate at least 
R and all decoding rules, when the signal power is P and the noise is D. One 
then defines 

£{R,P,V) = linisup£:(n,i?,P,X'), and 

n 

£{R,P,V) ^ liniinf f(n,i?,P,2?). 

n 

Assuming these are identical, one denotes this common limit by £{R, P,!)). 
Here is the relation between the notation of the Gaussian case: 

foxact(i?, A) = £{R, P, WGN(a2)) 

with A = P/a^. 

9.2.4 Connections between Error Exponents with and without Re- 
strictions 

Theorem 5-sen. Let V be a stationary and ergodic noise process and let a > 1. 
For all families of stationary and ergodic point processes = {m"},- eac/i with 
normalized logarithmic density bounded from above by —h(T>) — ln(a), for all 
families of jointly stationary decoding regions C = {C"} and for all P > such 
that i ln(27reP) > h{'D) + ln(a), we have 

£(^hi{2TTeP)-h{V)-\n{a),P,Vj > n{fi,C,a-,V), (119) 



and 



£( Cp{V) -\n{a) - ^Inl 1 + ^— ] ,P,V] > 7r(^, C, P), (120) 
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whe 



7r(/i, C, a—, 2?) = lim7r(/i, C, a — e, I?). 



wit/i n^fi^C, x,V) the error exponent without restriction defined in I162\}. In 
addition 



liminff (Cpp) - ln{a),P,V) > n{pL,C,a-,V). 



(121) 
□ 



Proof. From the very definition of Palm probabilities, for all n, 

.J2k S.t. T^eB"{Q,V^)P<^''^ 



g-n/i(X')g-nln(Q)y^(-^/^-) ' 



where pe,fe denotes the probability that TJ} + does not belong to given 
{r;",C;"}i. Hence, for all 7 > 0, 

pr(n,Ai",C",a,I?) 



> 



> 



^''=^'=-'-^"(B"(0,\/7rP))>(27reP)te-"''(»)e-"i"(°+^) 
k S.t. TJ^£B^{0,\/nP) 

g-nfi(P)g-nln(a)yg(|^/^-) 

1 ln(2^eP) - h{V) ln(a + 7), P, p)g-nin(a+,)gnin(o) (2jreP)^ 

2 V^{^nP) 



Hence 



— ln(pf(n,A.",C",a,I?)) 
< - i In (^Pe,opt (n, i ln(27reP) - h{V) - ln(a + 7), -P, 2?) 

-iln(p" (/i"(P"(0, V^)) > (27reP)^e-"''(^)e-"("+''))) 
- ln(a) + In (a + 7) - - In ( (^^^-^ ] . 
When taking a liminf in n, we get that 

7r(^, C, a, < ln(l + j/a)) ln(27reP) - - ln(a + 7), P, P 



(122) 



55 



Equivalently, for all a > 1 for all 7 small enough, 
7r(/x,C,a-7,X') < ln(5)/(a-7))+£ Q ln(27reP) - /i(2?) - ln(5),P,X'^ (123) 

and (|119p follows when letting 7 to in the last relation. 

We get (|120p from (|119p when using the second inequality of (|116p and the 
fact that the function x — > £_{x^ -P, 2?) is non-increasing. 

By the same monotonicity arguments, we get from (|122p that for P large 
enough and for 7 small enough, 

7r(^, C, a, 2?) < ln(l + 7/a)) + £ (^Cp(P) - ln(a + 7) - ^ In (^1 + ^) , P, P 
Hence, for all a > 1, for all P large enough and 7 small enough, 



TT I //,C, " ^ <ln(5/(5-7))+£(Cp(2?)-ln(5),P,P) 
1 + ^ 

and (|12ip follows by letting 7 tend to and P tend to 00. □ 
As a direct corollary of the last theorem, we have 

£_ [cp{V) - ln(a) - 1 In (^1 + , P, 2?^ > 7r(Poi, £(P), a-, I?) (124) 

£ (^Cp (2?) - ln(a)-i In (^1 + ^^,P, 2?^ > tt (Mat, £(!?), a-, 2?) (125) 

with £(2?) the MLE decomposition for 23. 

9.2.5 AWGN Channel 

Given A ~ P/a^ > and a > 1, consider i£'cxact(^ In ^"^^ ,A), where the 
function P 1— >■ f(P,y4.) is defined in subsection 19.1.21 Then, it follows from 
Theorem 5-sen that, for all a > 1, 



lim f^ln^^,^) > v{a,cT^). (126) 

with 77(a,cr^) the error exponent defined in Section [9.1.21 Hence for all a > 1, 



'1 1 + 

lim fexact ( In ^,A ] > ^(a) , (127) 



with 77(0;) the Poltyrev exponent (jlOSp . 
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10 Appendix 

10.1 Density of the Norm of the Gaussian Vector 

We recall that the T{6, v) density is of the form 



1 c^^^fl"^^"^^! ^ 



If (T = 1, the density of L>f H h Dl is r(l/2, n/2), that is 

J[x)-i,^oe 2«/2^ T{n/2)- 



Hence by the change of variable formula for densities, the density of -^Z Df + • • • + Df-^ 
at r is 

n"(r) - 1 ^..r"''"/^ ^ r"~^ ^ 
5iW-l.>oe ^Vs'^ r(n/2)- 

10.2 Symmetric Exponential Distribution 

The differential entropy of the symmetric exponential distribution of variance 
0-2 is h{D) = ln(y2ecr). 

We prove below that among the distributions on R such that the mean of 
the absolute value is fixed, the distribution with maximal entropy is symmetric 
exponential. 

The differential entropy /i of a real-valued random variable with density r 
and with mean absolute value t is r^h^ +T^h^ + h{T^) with h{x) = x\n{x) + 
(l-x) ln(l-a;), 

/o 
T{x)dx 
'OO 

and 

poo /-O 

h+ = - ln{T{x)/T+)T{x)/T+dx, h-=- ln{T{x)/T^)T{x)/T-dx. 

Jo J -oo 

Let 

POO pO 

= t{x)/t'^ xdx, = ^ t{x)/t~ xdx. 
Jo J -oo 

We have T+t+ +T^t^ = t. The maximum of entropy distribution among the set 
of distributions on with mean a is the exponential of parameter a. Hence, 
for fixed, the maximum entropy is reached for t{x)/t^ exponential of mean 
. Using a similar argument for t{x)/t~ , we get that the maximal entropy is 
obtained for the parameters t+ = v* and t+ = u* given by: 

{u*,v*) argmax„g[o i] „g[o i]wln(eu) + (1 - v)\n{e{t - uv)/{l - v)) + h{v). 

For fixed w, the optimal u is u* = t. At this value the function to be optimized 
in V is In(et) + h{v) which is maximal in v* — 1/2. 
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10.3 Direct Proof of Theorem 4-wgn-Poisson 

Theorem 4-wgn-Poisson is a special case of Theorem 4-sen-Poisson (see the 
WGN example fohowing the proof of Theorem 4-sen-Poisson). We nevertheless 
give a direct proof below. 

We first get an upper bound on 



9 , n"/2 1 



/E 2"/2 r(n/2)' 
Using the bound 



for some constant K > 1, wc get 



1 2 I ^/E 11 

< 



V^2"/2r(n/2) - 2"/2 0F(n/2e)«/2 0Fe-"/2 

Hence 



(129) 



The bounds (|128p can also be used to show that there exist constants C and 
C such that for all n large enough for all v > 0, 

{2e7Tv)^^^ < VSiV^) < {2env)^-^ (130) 
V 2 + n y'2 + n 

We now get an upper bound on 1 — p->^nVg(vcr^) ^ a; > 0, 1 — < 
min(x, 1). Hence 

1 _ g-A„y«"(t,av^) < mm{XnV]^{vay/7i), 1) 
and for n large enough, 

^=(-) <XnVEivaV^)<^=(-) . (131) 

Let a„ denote the real number such that 

XnVEianCrVn) = 1. 



For n large enough, a„ > a. In addition a„ — > a as ti ^ oo. Thus, from (|64p 
and (HSU, 

Pe(ri) < C' r"e-"(*-3-^'"(")+'"("))id7; 

Jo V 

^ r,-(^-^-M"))i^^^ (132) 
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The minimum of the function 

hi{v) = ^~^~2\n{v)+Ha) 

is reached for v = -\/2 whereas that of the function 

1 

is reached for v ^ I. 

If a > V2, the minimum of the function hi{v) is reached over the [0, a] 
interval and the dominant term of the first integral is 

g-n/ii(y2) ^ g-n(i~ln(2)+ln(a)) _ 

Since the function hi(v) is increasing on (1, oo], the dominant term in the second 
one is less than 

From (fT32l) . defining hi{v) = hi{v) - hi{V2) and /12(f) = h2{v) - hi{V2), we 
get 

-i ln(pe(n)) > Q - ln(2) + ln{a)^ + /3{n) 

with 

/3(n) = --ln -/ dv + ^ dv)]. 

n \Tr Jo V TT J^^ V J 

We now conclude the proof by showing that each of the positive functions 
h{n) = C / dv, h[n) = y— \ dv 

Jo V TT J^^ V 

is bounded from above by a constant when n tends to infinity. By dominated 
convergence, l2in) -^n^oo 0. For all < c < a/2, we have 

2r f -nhi(v) 

h{n)<C'^—+ dv, 

V2 - C J[0,a„]\[\/2-c,y2+c] 

where the last integral tends to by dominated convergence. This concludes 
the proof in this case. 

For a < -\/2, the dominant term of the first integral is 



Since the function h^iv) is increasing for w > 1, the dominant term of the second 
one is 
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We then proceed as above. From (|132p . defining hi{v) = hi{v) — hi{a) and 
h2{v) = h2[v) — h2{a), with 

1 

hi{a) =h2{a) ^ — - ln(a), 

we get 

with as above (with these new functions) and it is enough to show that 
each of the positive functions 

/•cIti -Ti/'ii(ti) rz: poo ~nh2(v) 

h{n) = C / dv, hin) = ^ / dv 

Jo V TT J^^ V 

is bounded from above by a function g{n) such that i hi(g(n)) tends to when 
n tends to infinity to complete the proof. For all < c < a, and for n large 
enough, 

h{n)<C'^+ / dv, 

a - C J[o,a„]\[Q-c,a+c] 

where the last integral tends to by dominated convergence. Similarly, for n 
large enough, 

hin) < + / dv, 

a-C J[a^,oo\\[a-c,a+c] V 

where the last integral tends to by dominated convergence. Hence one can 
take g{n) = Ky/n and this concludes the proof. □ 



10.4 Completion of the Proof of Lemma 1-mn 

We have to prove that if {An} and {Bn} are real- valued sequences that both sat- 
isfy an LDP with the same good and convex rate function /, then all sequences 
{Xn} such that 

An < Xn < Bn, Vn 

satisfy an LDP with the rate function / under the assumptions of the lemma. 

First apply Varadhan's integral lemma to conclude that for every real 9 we 
have 

lim - ln£;[e^^"] = A{9) = sup(6'a; - I{x)) 

n n X 

and the same for i?„. Notice that A{9) is necessarily lower-semicontinuous (since 
it is a Fenchel-Legendre transform). 

From this we see (by separately considering 9 > and 9 < 0, that X„ also 
satisfies this equation. But then, since we assumed that A is essentially smooth, 
by the Gartner-Elhs theorem (Part (c) of Thm. 2.3.6 on pg. 45 of [7|) {X„} 
satisfies an LDP with good and convex rate function the Fenchel-Legendre of 
A, namely / (since we assumed / to be convex). 
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10.5 A Property of Differential Entropy 

In this section the logarithm is taken to the natural base for concreteness. We 
adopt the usual convention that log § = 0. 



10.5.1 The wn case 

We make the general observation that for any two densities / and g on the real 
line, the negative part of the integral defining the relative entropy K{g\\f) is 
bounded above by - (this is true whether or not g is absolutely continuous with 
respect to /). Indeed, since we have xlogx > —\ for all x > 0, we can write 

{gix)\og^)-dx = [{g{x)l{f{x)>0)\og^)^dx 

/(x)(^l(/(x)>0)log^)_da; 



< 



/ f{x)-dx<- 
J e e 



Here the first step is because if f{x) = then g{x) log is either or oo. As, 
a corollary, if S denotes the support of /, we also have 

, , , , q(x) , , 1 

{g{x)\0g^)^dx<- . 

IS fix) e 

We also make the observation that for each real x 

gix)\og^^ = g{x)logg{x) ~ g{x)\ogf{x) . (133) 

This certainly holds if g{x) and f{x) are either both positive or both 0. If g{x) 
is positive while f{x) is then the left hand side and the right hand side both 
equal oo, while if g{x) is with f{x) positive, both sides equal 0. This covers 
all the cases. 

Let / be a probability density on the real line for which 

X ^ \ogf{x) 

is a bounded function on S, where S denotes the support of / (we do not need 
continuity of this mapping for the discussion here; continuity is only needed in 
order to apply the contraction principle). Let g be another probability density 
on the real line which is absolutely continuous with respect to / (i.e. 1(5(2;) > 
0) ^ l(/(a^) > 0)). We claim that the differential entropy of g is well defined. 
For this, first observe that the integral 

9{x) \og f{x)dx 
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is well defined. This is immediate because this integral can also be written as 



gix) log f{x)dx 

s 

(because of the assumed absolute continuity of g with respect to /) and since 
X log f{x) is bounded as a map from S to the real line, both the positive and 
the negative part of this integral are finite. 

Using Equation (|133p . we may now write, for each real x, 

igix)logg{x))_ < (5(x)logfM)_ + (5(a;)iog/(x))_ . 

fix) 

Integrating on both sides, we conclude that J g{g{x) log g{x))^dx < oo. Since 
{g{x)logg{x))-dx = / {g{x) log g{x))-dx , 



JS 

again by the assumed absolute continuity of g with respect to /. the differential 
entropy of g is well defined. 

At this point we have shown that the validity of (|77p in Lemma 1-wn implies 
the validity of ([75)1 . 

More generally now, let / be an arbitrary probability density on the real line 
and g another probability density that is absolutely continuous with respect to 
/ and for which K{g\\f) is finite. We claim that the differential entropy of g is 
well defined iff the integral 



9ix)logf{x)dx 

is well defined. For one direction, note that if J {g{x) log f{x))-dx < oo then 
Equation (|133p implies that J{g(x) log g{x))-dx < oo, so the differential entropy 
of g is well defined, while if J {g{x) log f{x))-dx ~ oo and / g{x)log f{x)dx is 
well defined we must have J{g{x) log f{x))+dx < oo, which, with Equation (|133p 
and the assumption that K{g\\f) is finite implies that J{g(x) logg{x))-f^dx < oo 
and so once again the differential entropy of g is well defined. For the con- 
verse, assume that the differential entropy of g is well defined. Then either 
J{— g{x) log g{x))+dx or g(a;) log (7(a:;))_c?2: must be finite. Suppose the lat- 
ter is the case. From Equation (|133p we can write, for each real x, 

{-g{x)logf{x))^ < (g(x)log^)_ + {-g{x)logg{x))_ . 

Integrating on both sides we conclude that J{— g{x) log f{x))-dx < oo, so 
J g{x) log f{x)dx is well defined. In the remaining case, suppose that 

{-g{x) log g{x))-dx = oo, 
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while / (—17(2;) \ogg{x))+dx < 00. We also have, from Equation (|133p . that 

{-g{x)\ogf{x)U < {9{x)\og^)+ + {^g{x)\ogg{x))+ . 

fix) 

We now use the assumption that K{g\\f) is finite to see, by integrating on 
both sides of this equation, that J {—g{x) log f{x))+dx < 00. Thus, once again 
/ g{x) log f{x)dx is well defined. 

10.5.2 Markov case 

The Markov version of Equation (|133p is 

QyVlxj 

b{x)q{y\x) log = b{x)q{y\x) log q{y\x) - b{x)q{y\x) logp{y\x) . (134) 

p(y\x) 

As above, we can show that if 

{x,y) logp{y\x) 

is a bounded function on 5, where S denotes the support of a(x)p{y\x) , and 
if b{x)q{y\x) is a stationary Markov probability density on M.'^ which is abso- 
lutely continuous with respect to a{x)p{y\x), then the differential entropy rate 
of 6(x)g(y|x), i.e. the integral 

- J b{x)q{y\x) log q{y\x)dxdy 

is well defined. 

This allows us to move from the validity of (j8ip in Lemma 1-mn to the 
validity of ((82)) . 

More generally, as above we can show that if a{x)p{y\x) is an arbitrary sta- 
tionary Markov probability density on and b{x)q{y\x) is another stationary 
Markov probability density on that is absolutely continuous with respect to 
a(x)p{y\x) and for which the relative entropy rate K {b{x)q{y\x)\\a(x)p{y\x)) , 
i.e. the integral 

j b{x)q{y\x) log'^^^^dxdy 
is finite, then the differential entropy rate of b{x)q{y\x), i.e. the integral 

j Kx)q{y\x) log q{y\x)dxdy 
is well defined iff the integral 

j b{x)q{y\x) log p{y\x)dxdy 

is well defined. 

In both cases, all that is needed is to use Equation (|134p everywhere that 
we had earlier used Equation (|133p and to mimic the preceding proofs line by 
line. 
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10.6 Differential Entropy Counter— Example 

Below, we construct an example of a pair of probability densities g and / such 
that g has finite differential entropy, the relative entropy K{f\\g) is finite, but 
/ has undefined differential entropy. 

First consider (a„, n > 3) and (u„, n > 3), sequences of nonnegative numbers 
given by 

1 

— 1 1 — 7j To" 

n(log nj^ 

where the logarithm is to the natural base, for concreteness. Observe that 

a„M„ < oo , 

n 

while 

^ a„«„ log i- = ^ ^^^^ (log n + 2 log log n) = oo . 

Thus, we can find a nonnegative scaling constant a > such that the piccewise 
constant density /+ taking values au„ on disjoint intervals of length a„, for 
n > 3, integrates out to 1 and has differential entropy oo. Next, consider 
sequences (6n, n > 3) and n > 3) given by 

1 

bn 



n^(logn)^ 
Observe that 

bnWn < oo , 

n 

while ^ ^ 

V6„M;„log — = -(V -^nlogn) = -oo . 

Wn ^-^ n^(lognj^ 

Thus, we can find a nonnegative scaling constant 7 > such that the piecewise 
constant density /_ taking values "fWn on disjoint intervals of length 6„, for 
n > 3, integrates out to 1 and has differential entropy —00. The supports of 
the densities /+ and /_ can be made disjoint, so the density + i/_ has 
undefined differential entropy. 

Wc now construct a pair of probability densities g and / with the announced 
properties. First take {an,n > 3) and (w„,ri > 3) as above, and let {vn,n > 3) 
be defined by 

1 

" n(logn)'^ 

Observe that 

anVn < 00 , 

n 

and ^ ^ 

Y]a„v„log — = — -^(logn + 31oglog?i) < 00 , 
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and also 



a„w„ log — = — log log n < cx) . 

„ Vn ^ n{\ogny 



We can therefore choose a scaling constant (3 > such that the piecewise con- 
stant density taking values /3w„ on the same disjoint intervals of length a„ for 
n > 3 as those supporting /+ integrates out to 1, has finite differential entropy, 
and has K{f+\\g+) < oo. Next, take (&„,n > 3) and {wn,n > 3) as above, and 
let {xn, n > 3) be defined by 



logn 



Observe that 



E 



bnXn = > 72 1 < OO 

^-^ n^(logn)^ logri 



and ^ ^ 

V6„x„log — = E^Tl ^1 (log log n - log n) < oo , 

n 



n^(log n)^ log n 

and 

\ h in Incr 

Xn (log n)2 



E6„W„ log — = -^r-^ 77 log log n<(X) 



Wc can therefore choose a scaling constant (5 > such that the piecewise con- 
stant density taking values 8xn on the same disjoint intervals of length 6„ for 
n > 3 as those supporting /_ integrates out to 1. has finite differential entropy, 
and has if(/_||g_) < cx). If we define g = igr^ -I- \g-^ then is a probability 
density with finite differential entropy, for which i^(/||(7) is finite when / is 
the probability density defined earlier, but we saw already that the differential 
entropy of / is undefined. 



10.7 Proof of Equations (fT05|) - (|T06l ) 

If f < f , then 

so that c(u) = 0. 

For ^ < w < we have to find an upper bound on the volume of the 
portion of the ball of radius va^Jn around the point at distance va^/n from the 
origin (along some ray) that is outside the ball i?"(0, acr-v/n) (this is depicted 
by the shaded area on Figure [1]). A first upper bound on this volume is the 
portion of the former ball by the hyperplane perpendicular to the ray and at a 
distance da^Jn from it (i.e. a distance of (v + d)ay/n along this ray from the 

2 

origin) where d = ^ — u by elementary geometry. The latter portion is in turn 
included in a ball of radius a\fn\/ v'^ — (that is depicted by the dashed circle 
in Figured]). Hence, c{v) = -J%p^^^ . 
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Figure 1: The origin of the plane is the tagged codeword. The large ball centered 
on the origin is the exclusion ball of the Matcrn construction around the tagged 
codeword. Its radius is (a — t)(7\/n. The point X is the location of the noise 
added to the tagged codeword. Its norm is va^Jn. The ball centered on X with 
radius vcjy/n is the vulnerability region in the Poisson case. In the Matern ease, 
the vulnerability region is the shaded lune depicted on the figure. We are here 
in the case with ^ < v < . We upper bound the area of this lune by that of 

the ball of radius c = ^yri(v^~~dF)a with d as above. This ball is depicted by 
the dashed line disc. 

10.8 Proof of ( fTT2l) 

We have 

sup W(x",w) > 
with a;"(w) = {van/V^, 0, . . . , 0). The region 

includes the region 
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Figure 2: The origin of the plane is the tagged codeword. The large ball centered 
in and passing through point A is that with radius """t"^*^) ^ Point V is that 

with coordinate a;"(u) = (van/ a/2, 0, . . . , 0). The region R{n) is depicted by the 
union of the dashed region and the grey one. The volume V{n) is that of the 
grey region. 



This region is comprised of 2" ^ copies (one for each configuration of signs of 
the variables y2, ■ ■ ■ ,yn, see Figure [2]) of the following basic region: 

{™„ van (a — e)an 2van 1 

y" ^ {yi,...,y^)eRl:yi > -j=-, ^ < + ^» < ^| ' 

The volume V(n) of this basic region is the same as that of 

{„ (a - e - v)an -J^ van \ 
2;" = (yi,...,2/„)eM!^:^ _^<^y^<_|. 

namely 2~" times the volume of the Li ball of center and radius deprived 
of the Li ball of center and radius iSzzl^^l^IL^ that is 

V{n) = 2-" {V2var— - iV2{a - e - . 

\ n\ n\ J 

Hence 

- In I sup W(x'\v) I > - In (2""V(n)) ^„^oo ln(V2vea). 
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^ In ( sup M^(a;",w) ) < - In ( ¥0155^(0, ^) ) -^n^oo \n{V2vea). 



n 



^/2 



This completes the proof of (|112p . 
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